Power distribution panel display method and device based on computer vision

Abstract

The application provides a computer vision-based power distribution screen display method and device, which comprises the following steps: acquiring multi-angle entity images of a power distribution screen by using a camera device; extracting matching feature points in the multi-angle entity images, and mapping the matching feature points to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device; performing point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud; constructing a static model of the power distribution screen according to the dense point cloud; and displaying the running information of the power distribution screen obtained in real time to the static model of the power distribution screen. Through the above method, the running data of the power distribution screen is mapped to the static model of the power distribution screen in real time, the remote intelligent monitoring and management level is improved, the power grid operation decision is optimized, and the asset digitization of the power distribution screen is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital twinning, and particularly relates to a power distribution panel display method and device based on computer vision. BACKGROUND

[0002] In the process of new power system construction and digital transformation, power distribution panels and other power communication equipment play an irreplaceable role as the basic infrastructure of the power industry. In the operation and maintenance business scenarios of communication equipment, the demand for digitalization of communication equipment assets is increasingly high. There are a large number of mechanical, repetitive, and time-consuming operations in communication production line equipment, which are difficult to control and have high operation risks. At present, it is urgent to carry out research on the digitalization of power communication assets to improve the efficiency of overall management. SUMMARY

[0003] The present application provides a power distribution panel display method and device based on computer vision, which maps the operation data of the power distribution panel into the static model of the distribution panel in real time, improves the level of remote intelligent monitoring and management, optimizes power grid operation decision-making, and realizes the digitalization of power distribution panel assets.

[0004] The present application provides a power distribution panel display method based on computer vision, comprising:

[0005] acquiring multi-angle entity images of the power distribution panel using a camera device;

[0006] extracting matching feature points in the multi-angle entity images, and mapping the matching feature points to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device;

[0007] performing point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud;

[0008] constructing a static model of the power distribution panel according to the dense point cloud;

[0009] displaying the real-time acquired operation information of the power distribution panel in the static model of the power distribution panel.

[0010] According to the power distribution panel display method based on computer vision provided by the present application, the matching feature points in the multi-angle entity images are extracted, and the matching feature points are mapped to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device, comprising:

[0011] identifying the power distribution panel region in the multi-angle entity images;

[0012] segmenting the power distribution panel region according to the GrabCut algorithm to obtain a foreground image of the power distribution panel;

[0013] Extract dense feature points in the foreground image, and perform feature point pair matching on the dense feature points to obtain matching feature points;

[0014] Remove the mismatched feature points in the matching feature point pairs;

[0015] According to the spatial coordinates of the camera equipment, the matching feature points with the removed mismatched feature points are mapped into a three-dimensional sparse point cloud.

[0016] According to the present application, a computer vision-based power distribution screen display method is provided, which performs point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud, comprising:

[0017] Obtain the depth map of the multi-angle entity image at different angles;

[0018] According to the image field information obtained from the depth map, perform patch expansion and patch filtering on the three-dimensional sparse point cloud to obtain the dense point cloud.

[0019] According to the present application, a computer vision-based power distribution screen display method is provided, which constructs a power distribution screen static model of the power distribution screen according to the dense point cloud, comprising:

[0020] According to the light speed adjustment method, constrain the re-projection error of the dense point cloud, and according to the Poisson surface reconstruction algorithm, perform curved surface reconstruction on the dense point cloud to obtain the power distribution screen static model.

[0021] According to the present application, a computer vision-based power distribution screen display method is provided, which extracts dense feature points in the foreground image, and performs feature point pair matching on the dense feature points to obtain a plurality of matching feature points, comprising:

[0022] The harris corner detection algorithm is used to extract the dense feature points in the foreground image;

[0023] The KLT algorithm is used to track and match the dense feature points to obtain a plurality of matching feature points.

[0024] According to the present application, a computer vision-based power distribution screen display method is provided, which further comprises the following steps before extracting the dense feature points in the foreground image:

[0025] The emphasize operator is used to enhance the image contrast of the foreground image.

[0026] According to the present application, a computer vision-based power distribution screen display method is provided, which removes the mismatched feature points in the matching feature points, comprising:

[0027] Select a preset number of sample feature points from the matching feature points, and the preset number of sample feature points are not collinear;

[0028] According to the preset number of sample feature points, a matrix model is constructed.

[0029] Calculate the projection error of each matching feature point and the matrix model.

[0030] If the projection error is greater than or equal to a preset error threshold, the matching feature point is removed as a mismatched feature point.

[0031] According to the present application, a computer vision-based power distribution screen display method is provided, which further comprises:

[0032] The SURF algorithm is used to extract sparse feature points in the foreground image of the power distribution screen to obtain the spatial coordinates of the camera device.

[0033] The present application also provides a computer vision-based power distribution screen display device, comprising:

[0034] An image acquisition unit is configured to acquire multi-angle entity images of the power distribution screen.

[0035] A feature point matching unit is configured to extract matching feature points in the multi-angle entity images and map the matching feature points to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device.

[0036] A feature point processing unit is configured to perform point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud.

[0037] A model construction unit is configured to construct a power distribution screen static model of the power distribution screen according to the dense point cloud.

[0038] An information display unit is configured to display the real-time acquired operation information of the power distribution screen to the power distribution screen static model.

[0039] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the computer vision-based power distribution screen display method as described above.

[0040] The present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to realize the computer vision-based power distribution screen display method as described above.

[0041] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the computer vision-based power distribution screen display method according to any one of the above.

[0042] The application provides a computer vision-based power distribution screen display method and device, which constructs a static model of a distribution screen through computer vision technology, and integrates real-time running information of the power distribution screen into the static model of the distribution screen, thereby improving the intuitiveness and efficiency of operation and maintenance management. The computer vision-based power distribution screen display method and device realize acquisition, processing and display of multi-angle entity images of the power distribution screen, and improve the digital level of the power distribution screen. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0044] Figure 1 Fig. 1 is one of flow diagrams of a computer vision-based power distribution screen display method provided by the application.

[0045] Figure 2 Fig. 2 is a schematic diagram of the principle of running information recognition of a power distribution screen.

[0046] Figure 3 Fig. 3 is another flow diagram of a computer vision-based power distribution screen display method provided by the application.

[0047] Figure 4 Fig. 4 is a schematic diagram of the principle of obtaining a three-dimensional sparse point cloud based on the triangulation principle.

[0048] Figure 5 Fig. 5 is a third flow diagram of a computer vision-based power distribution screen display method provided by the application.

[0049] Figure 6 Fig. 6 is a fourth flow diagram of a computer vision-based power distribution screen display method provided by the application.

[0050] Figure 7 Fig. 7 is a schematic diagram of a static model of a distribution screen.

[0051] Figure 8 Fig. 8 is a result diagram of a computer vision-based power distribution screen display device provided by the application.

[0052] Figure 9It is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0054] The terms "first", "second", and the like in the specification of the present application and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or chronological sequence. It should be understood that the descriptions used in this way can be interchanged under appropriate circumstances, so that the embodiments can be implemented in an order other than that illustrated or described in the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or modules does not have to be limited to those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps appearing in the present application does not mean that the steps in the method flow must be executed in the order / time sequence indicated by the naming or numbering. The execution order of the steps named or numbered can be changed according to the technical purpose to be achieved. The division of modules appearing in the present application is a logical division, and in actual application, there can be another division manner, for example, a plurality of modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the modules shown or discussed can be through some interface, the indirect coupling or communication connection between the units can be electrical or other similar forms, which are not limited in the present application. In addition, the modules or sub-modules described as separate components can or can not be physically separated, can or can not be physical units, or can be distributed to a plurality of circuit units, and some or all units can be selected according to actual needs to achieve the purpose of the present application.

[0055] The specific content of the present application will be described below. Figures 1-9 The specific content of the present application will be described below.

[0056] Figure 1 One of the flow schematic diagrams of a computer vision-based power distribution screen display method provided by an embodiment of the present application, comprising steps S101-S105.

[0057] Step S101: Use a camera device to acquire multi-angle physical images of the power distribution panel.

[0058] In embodiments of the present invention, the imaging device includes, but is not limited to, a camera, which is used to acquire multi-angle physical entity images of the power distribution panel to be reconstructed, forming an initial image dataset.

[0059] Step S102: Extract matching feature points from multi-angle entity images and map the matching feature points into a three-dimensional sparse point cloud based on the spatial coordinates of the camera device.

[0060] Step S103: The 3D sparse point cloud is densified to obtain a dense point cloud.

[0061] Step S104: Construct a static model of the power distribution panel based on the dense point cloud.

[0062] Step S105: Display the real-time acquired operating information of the power distribution panel in the static model of the power distribution panel.

[0063] The following describes the process of acquiring the operation information of the power distribution panel in an embodiment of the present invention.

[0064] In an embodiment of the present invention, a camera is used to capture color images of the front of the power distribution panel in real time at regular intervals. Based on the Faster R-CNN deep learning network, objects such as switches, terminals, text, and status lights are identified and their interest boxes are extracted from the real-time images. The anchor boxes of each object are analyzed to obtain the operating information of the power distribution panel.

[0065] For example, a certain number of power distribution panel images are acquired, and the labelimg toolbox is used to create an image training set by anchoring objects that represent the operating status, such as switches, terminals, text, and status lights. The above image training set is then input into the Faster R-CNN network for training. Finally, the image set to be recognized is input, and the Faster R-CNN network is used to recognize the operating status of each object, such as switches, terminals, text, and status lights.

[0066] like Figure 2 As shown, for an image with a text region selected by an anchor box of interest, OCR technology is used to extract and recognize the text to accurately obtain the working status of the power distribution panel. Specifically, this includes: using a convolutional recurrent neural network to extract image text features and recognize the text content; using a connection-time classifier to merge duplicate characters and remove empty characters, and outputting the recognized text result.

[0067] Finally, the text results are displayed as the operation information of the power distribution panel in the static model of the power distribution panel.

[0068] Figure 3Flowchart two of a computer vision-based power distribution screen display method according to an embodiment of the present application.

[0069] In one possible implementation, in step S102, matching feature points in the multi-angle entity image are extracted, and the matching feature points are mapped to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device, as shown in Figure 3 The method comprises steps S301-S305.

[0070] In step S301, the power distribution screen region in the multi-angle entity image is identified.

[0071] In an embodiment of the present application, the Faster R-CNN network is used to identify the power distribution screen in the initial image to extract the anchor frame region image, i.e., the power distribution screen region. Specifically, the labelimg toolbox is used to anchor the region occupied by the multi-angle entity image to make an image training set of the power distribution screen; the image training set is input into the Faster R-CNN network for training; and the multi-angle entity image in the image set to be identified is framed by the Faster R-CNN network to identify the power distribution screen region.

[0072] In step S302, the power distribution screen region is segmented according to the GrabCut algorithm to obtain a foreground image of the power distribution screen.

[0073] Specifically, the GrabCut algorithm is used to perform semantic segmentation on the image in the anchor frame to delete the background and obtain the foreground image of the power distribution screen.

[0074] Specifically, the GrabCut algorithm is used to perform semantic segmentation on the image in the anchor frame to delete the background and obtain the foreground image of the power distribution screen.

[0075] E=(α,k,θ,z)=D(α,k,θ,z)+V(α,z)

[0076] wherein, α is a transparency coefficient, the background is set to 0 and the foreground is set to 1; k∈{1,2,···,K} is the number of GMM models; θ=(μ,δ2) is the mean and covariance of GMM; z represents a pixel; D represents the matching degree of the transparency coefficient α and the pixel z, which takes the minimum value when correctly classified; V is a boundary term, which reflects the similarity of adjacent pixels and takes the minimum value at the image boundary.

[0077] Step S303, extracting dense feature points in the foreground image, and performing feature point pair matching on the dense feature points to obtain matched feature point pairs.

[0078] Step S304, eliminating the mis-matched feature points in the matched feature point pairs.

[0079] Step S305, mapping the matched feature points with the mis-matched feature points eliminated into a three-dimensional sparse point cloud according to the spatial coordinates of the camera equipment.

[0080] In the embodiments of the present application, the harris corner detection algorithm is used to extract dense feature points, and the KLT tracking algorithm is used to complete the matching of the feature point pairs, and then the Ransac random consistent sampling algorithm is used to eliminate the mis-matched feature points. Finally, based on the triangulation principle, as shown in Figure 4 , the three-dimensional spatial coordinates of the feature points can be calculated by using the spatial coordinates of the camera and the pixel coordinates of the matched feature points, and the three-dimensional sparse point cloud is mapped. For details, refer to the following embodiment content.

[0081] In a possible implementation, extracting dense feature points in the foreground image, and performing feature point pair matching on the dense feature points to obtain matched feature point pairs, includes steps S303A and S303B.

[0082] Step S303A, using the harris corner detection algorithm to extract dense feature points in the foreground image.

[0083] Specifically, a window function ω(x, y) with a center as a coordinate (x, y) is used as a window to slide in the x and y directions of the foreground image, the pixel gray value change E(u, v) in the window is calculated, and a two-dimensional Taylor formula expansion is performed on it; the corner response function H = min(λ1, λ2) in the window is calculated; wherein H is the corner response function; λ1 and λ2 are eigenvalues of the matrix. According to the threshold value threshold set in advance, it is determined that the pixel point is a dense feature point when H > threshold.

[0084] Step S303B, using the KLT algorithm to track and match the dense feature points to obtain matched feature point pairs.

[0085] Specifically, the KLT algorithm is used to track and match the dense feature points between images, and whether there is a matched feature point pair in the window is judged by comparing the square of the pixel gray difference in the preset window of the two foreground images; the implementation process is as follows:

[0086] 1. Detecting feature points of the first picture.

[0087] 2. Estimating the motion trajectory by translating or mapping each feature point in the second picture.

[0088] 3. Track the dense feature points according to the motion trajectory.

[0089] Wherein, before tracking the features, a square feature window W with a side length of h is drawn around the dense feature point. Define the corresponding points of the same point in the two foreground images as (ux, uy) and (ux+dx, uy+dy), wherein dx and dy are the image motion direction, i.e. the change direction of the object. If the two points match, then in the window W, the foreground image A has a window of e-h / 2, and the foreground image B has a window of e+h / 2, and the minimum square difference of the gray scale. The offset of the dense feature point is d=[dx, dy]T, the projection point is e=[ux, uy]T, and the square difference of the gray scale is ε, which is expressed by the integral expression as follows:

[0090]

[0091] Wherein, I(e) and J(e) are the gray scale values of the images I and J, i.e. the gray scale values of the image 1 and the image 2; and w(e) is the function of the window W. Take the partial derivative of the above formula with respect to d, and when the result is 0, the minimum value of ε is obtained.

[0092] In a possible implementation, before extracting the dense feature points in the foreground image, the method further includes: enhancing the image contrast of the foreground image by using an emphasize operator.

[0093] Figure 5 A flowchart of a computer vision-based power distribution screen display method according to an embodiment of the present application.

[0094] In a possible implementation, in step S304, the false matching feature points in the matching feature points are removed, as shown in the following formula: Figure 5 The method includes steps S501-S504.

[0095] In step S501, a preset number of sample feature points are selected from the matching feature points, and the preset number of sample feature points are not collinear.

[0096] In step S502, a matrix model is constructed according to the preset number of sample feature points.

[0097] For example, 4 sample points are randomly selected from the matching feature point pair set, the sample points are not collinear, and then a transformation matrix is calculated and recorded as the matrix model N.

[0098] In step S503, the projection error of each matching feature point and the matrix model is calculated.

[0099] For example, the distance between the matching point and the matrix model N is calculated, and the mean value of the calculated distance is taken as a threshold Tr.

[0100] Step S504, if the projection error is greater than or equal to the preset error threshold, the matching feature point is removed as a mismatched feature point.

[0101] For example, the projection error of all points in the matching feature point pair set and the model N is calculated, and if the error is less than the threshold Tr, the point is added to the feature point set Bn. If the projection error is greater than or equal to the preset error threshold, the matching feature point is removed as a mismatched feature point.

[0102] In an embodiment of the present application, if the number of elements of Bn is greater than the optimal number Bn_best, let Bn_best = Bn, and update the iteration number k B , which can be calculated by the following formula:

[0103]

[0104] Where p is the confidence, generally 0.995, n is the proportion of the feature point set, and z is the sample point number, which is 4. When the iteration number is greater than kB, the process ends, otherwise the iteration number is increased by one, and the above steps S501-S504 are repeated.

[0105] Figure 6 A flowchart of a computer vision-based power distribution screen display method according to an embodiment of the present application.

[0106] In one possible implementation, in step S103, the three-dimensional sparse point cloud is densified to obtain a dense point cloud, as shown in the following formula: Figure 6 The process includes steps S601-S602.

[0107] Step S601, obtaining a depth map of the multi-angle entity image at different angles.

[0108] In an embodiment of the present application, the depth map is an image or an image channel, which contains the distance information of each point in the scene to the observation point, and is usually represented by gray scale, wherein the brighter pixel represents the object closer to the observation point, and the darker pixel represents the object farther away from the observation point.

[0109] Step S602, according to the image field information obtained from the depth map, the three-dimensional sparse point cloud is expanded and filtered to obtain a dense point cloud.

[0110] Initialization of matching features: extracting feature points in the image, for the feature points in the I-angle entity image, finding matching feature points in other angle entity images to form matching point pairs, and using triangulation to form three-dimensional space points.

[0111] Patch expansion: the three-dimensional space points formed above are sorted according to the distance from the optical center O(I) of the first angle entity image from small to large, and are expanded one by one until the patches are generated, and the center cI(p), normal vector nI(p) and corresponding reference image RI(p) of the patch are initialized;

[0112] Patch filtering: the discontinuous patches that do not meet the consistency standard are filtered by three filters.

[0113] The above steps are repeatedly iterated to obtain the dense point cloud finally by patch expansion and patch screening of the point cloud.

[0114] In a possible implementation, in step S103, the power distribution screen static model is constructed according to the dense point cloud, including: the re-projection error of the dense point cloud is constrained according to the light speed adjustment method, and the dense point cloud is reconstructed into a curved surface according to the Poisson surface reconstruction algorithm to obtain the power distribution screen static model.

[0115] Specifically, based on the light beam adjustment method theory, the Levebberg-Marquardt algorithm in the least square method is used to minimize the error and reduce the re-projection error, as shown in the following formula:

[0116]

[0117] Wherein, K represents the number of space points, L represents the number of camera angles, x ij represents the camera plane projection of the i-th calculated three-dimensional point under the j-th angle, Q(P j ,X i ) represents the predicted projection point of the space three-dimensional point X i to the camera P j plane, and d(x,y) represents the Euclidean distance between the image points x and y. The Poisson surface reconstruction algorithm is used to reconstruct the dense point cloud into a power distribution screen static model as shown in Figure 7 .

[0118] In a possible implementation, the power distribution screen display method based on computer vision provided by the application further includes: extracting sparse feature points in the foreground image of the power distribution screen by using the SURF algorithm to obtain the spatial coordinates of the camera equipment.

[0119] Specifically, sparse feature points in the foreground image are extracted by using the SURF algorithm, and for each sparse feature point, a 64-dimensional descriptor is formed based on the harr wavelet feature; the specific implementation process is as follows:

[0120] 1. Constructing Gaussian scale space and calculating Hessian matrix H(x, y, σ). Given a point (x, y) in the foreground image, the Hessian matrix H(x, y, σ) at scale σ is expressed as:

[0121]

[0122] where represents the convolution of the foreground image and the Gaussian second-order differential at (x, y), L xy (x, y, σ), L yy (x, y, σ) is similar.

[0123] 2. Comparing each pixel point processed by the Hessian matrix with 26 points in the two-dimensional image space and the scale space neighborhood, preliminarily locating the key points, and then filtering out the key points with weak energy and the incorrectly located key points to screen out the final stable sparse feature points.

[0124] 3. Determining the main direction distribution of the sparse feature points and generating a 64-bit descriptor of the feature points.

[0125] Subsequently, the Euclidean distance of the sparse feature points between the foreground images is calculated, and the feature point pairs less than the threshold value are matched by comparison; specifically, the nearest distance d1 and the second nearest distance d2 between any two features in different pictures are calculated, and when the ratio of the nearest distance d1 to the second nearest distance d2 is less than a certain threshold value, it is determined to be matched. The expression of the distance L is:

[0126]

[0127] where x i , y i are the i-th bits of the 64-bit descriptors of different features, respectively.

[0128] Finally, based on the successfully matched sparse feature points (i.e., the matched feature points), the intrinsic matrix is calculated using the five-point method to obtain the camera pose, i.e., the camera pose, and then the spatial coordinates of the camera device are obtained; the specific implementation steps are as follows:

[0129] 1. Randomly selecting five pairs of matched feature points to calculate the intrinsic matrix E, as shown in the following formula:

[0130]

[0131] where x1, y1 and x2, y2 are the coordinates of a matched feature point pair in two foreground images, respectively.

[0132] 2. Singular value decomposition of the intrinsic matrix E to calculate the rotation matrix R and the translation matrix T.

[0133] 3、By seeking the matching feature points between the foreground images and combining the corresponding relationship between the coordinate systems, the world coordinates when the camera captures the image, i.e., the spatial coordinates of the camera equipment, can be calculated, as shown in the following formula:

[0134]

[0135] In the formula, M1 represents the intrinsic matrix of the camera, which is obtained by Zhang Zhengyou calibration method; M2 represents the extrinsic matrix of the camera, which is mainly obtained by singular value decomposition of the intrinsic matrix sought above.

[0136] The acquisition of the intrinsic matrix of the camera in the embodiment of the application will be described below. In the embodiment of the application, the multi-angle entity image can be input into the Camera Calibrator toolbox in the Matlab software, and the intrinsic parameters of the camera, i.e., distortion parameters, camera focal length, etc., are calculated and obtained, so as to obtain the intrinsic matrix of the camera.

[0137] The application can achieve at least one of the following beneficial effects by using the above method:

[0138] 1. By computer vision technology, a static model of the power distribution screen is constructed, and real-time running information of the power distribution screen is integrated into the static model of the power distribution screen, thereby improving the intuitiveness and efficiency of operation and maintenance management. The acquisition, processing and display of the multi-angle entity image of the power distribution screen are realized, and the digital level of the power distribution screen is improved.

[0139] 2. By recognizing, segmenting and extracting feature points of the power distribution screen region of the multi-angle entity image, and eliminating the mis-matching feature points, the accuracy and quality of the three-dimensional sparse point cloud are improved, and the accuracy and reliability of subsequent three-dimensional modeling are ensured.

[0140] 3. By using the depth map information, the densification of the three-dimensional sparse point cloud is realized, and the detail richness of the power distribution screen model is further improved.

[0141] 4. The light speed adjustment method and the Poisson surface reconstruction algorithm are used to effectively constrain the re-projection error of the dense point cloud, and high-precision curved surface reconstruction is realized, thereby improving the accuracy and practicality of the static model of the power distribution screen.

[0142] 5. By combining the Harris corner detection algorithm and the KLT algorithm, the effective extraction and matching of the dense feature points in the foreground image are realized, and the accuracy and stability of the feature point pairs are improved.

[0143] 6. Before extracting the dense feature points, the contrast of the foreground image is enhanced by the emphasize operator, the image quality is improved, and the accuracy and reliability of the feature point extraction are further improved.

[0144] 7. By the method of constructing a matrix model and calculating a projection error, the mismatched feature points are effectively eliminated, and the accuracy and quality of the three-dimensional sparse point cloud are further improved, thereby providing a more reliable data basis for subsequent three-dimensional modeling and operation and maintenance management.

[0145] The computer vision-based power distribution screen display device provided by the application is described below, and the computer vision-based power distribution screen display device described below can be correspondingly referred to the computer vision-based power distribution screen display method described above.

[0146] Figure 8 The structural schematic diagram of the computer vision-based power distribution screen display device provided by the application comprises:

[0147] The image acquisition unit 810 is configured to acquire multi-angle entity images of the power distribution screen.

[0148] The feature point matching unit 820 is configured to extract matching feature points in the multi-angle entity images and map the matching feature points to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device.

[0149] The feature point processing unit 830 is configured to perform point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud.

[0150] The model construction unit 840 is configured to construct a distribution screen static model of the power distribution screen according to the dense point cloud.

[0151] The information display unit 850 is configured to display the real-time acquired operation information of the power distribution screen to the distribution screen static model.

[0152] In a possible implementation, the feature point matching unit 820 comprises:

[0153] The region identification subunit is configured to identify the distribution screen region in the multi-angle entity images.

[0154] The image segmentation subunit is configured to segment the distribution screen region according to the GrabCut algorithm to obtain a foreground image of the power distribution screen.

[0155] The feature point matching subunit is configured to extract dense feature points in the foreground image and perform feature point pair matching on the dense feature points to obtain matching feature point pairs.

[0156] The feature point elimination subunit is configured to eliminate mismatched feature points in the matching feature point pairs.

[0157] The feature point mapping subunit is configured to map the matching feature points, from which the mismatched feature points are eliminated, to the three-dimensional sparse point cloud according to the spatial coordinates of the camera device.

[0158] In a possible implementation, the feature point processing unit 830 includes:

[0159] The depth map acquisition subunit is configured to acquire depth maps of the multi-angle entity image at different angles.

[0160] The dense point cloud acquisition subunit is configured to perform patch expansion and patch filtering on the three-dimensional sparse point cloud according to the image field information acquired from the depth map, to obtain a dense point cloud.

[0161] In a possible implementation, the model construction unit 840 is specifically configured to constrain the re-projection error of the dense point cloud according to a light speed adjustment method, and perform curved surface reconstruction on the dense point cloud according to a Poisson surface reconstruction algorithm, to obtain the static model of the power distribution panel.

[0162] In a possible implementation, the feature point matching subunit is configured to extract dense feature points in the foreground image by using a Harris corner point detection algorithm, and track and match the dense feature points by using a KLT algorithm, to obtain a matching feature point pair.

[0163] In a possible implementation, the feature point matching unit 820 further includes an image enhancement subunit configured to enhance the image contrast of the foreground image by using an emphasize operator.

[0164] In a possible implementation, the feature point elimination subunit is specifically configured to select a preset number of sample feature points from the matching feature points, the preset number of sample feature points are not collinear, construct a matrix model according to the preset number of sample feature points, calculate the projection error of each matching feature point and the matrix model, and eliminate a matching feature point as a mismatching feature point if the projection error is greater than or equal to a preset error threshold.

[0165] In a possible implementation, the power distribution panel display device based on computer vision further includes a camera device coordinate acquisition unit configured to extract sparse feature points in a foreground image of the power distribution panel by using a SURF algorithm, to obtain the spatial coordinates of the camera device.

[0166] Figure 9 An entity structure schematic diagram of an electronic device is shown in FIG. 1. Figure 9 As shown in FIG. 1, the electronic device can include a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 complete mutual communication through the communications bus 940. The processor 910 can invoke a logical instruction in the memory 930 to execute a power distribution panel display method based on computer vision, which includes:

[0167] Obtaining multi-angle entity images of the power distribution panel.

[0168] Extracting matching feature points in the multi-angle entity images, and mapping the matching feature points into a three-dimensional sparse point cloud according to spatial coordinates of the camera equipment.

[0169] Carrying out point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud.

[0170] Constructing a power distribution panel static model of the power distribution panel according to the dense point cloud.

[0171] Displaying running information of the power distribution panel obtained in real time into the power distribution panel static model.

[0172] In addition, the logic instructions in the memory 930 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0173] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the computer vision-based power distribution panel display method provided by the above-mentioned methods, and the method comprises:

[0174] Obtaining multi-angle entity images of the power distribution panel.

[0175] Extracting matching feature points in the multi-angle entity images, and mapping the matching feature points into a three-dimensional sparse point cloud according to spatial coordinates of the camera equipment.

[0176] Carrying out point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud.

[0177] Constructing a power distribution panel static model of the power distribution panel according to the dense point cloud.

[0178] Displaying running information of the power distribution panel obtained in real time into the power distribution panel static model.

[0179] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the computer vision-based power distribution panel display method provided by the above method, the method comprising:

[0180] obtaining multi-angle entity images of the power distribution panel.

[0181] extracting matching feature points in the multi-angle entity images, and mapping the matching feature points to a three-dimensional sparse point cloud according to the spatial coordinates of the camera device.

[0182] performing point cloud densification on the three-dimensional sparse point cloud to obtain a dense point cloud.

[0183] constructing a distribution panel static model of the power distribution panel according to the dense point cloud.

[0184] displaying the running information of the power distribution panel obtained in real time to the distribution panel static model.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0186] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus the necessary general hardware platform, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A computer vision-based method for displaying power distribution panels, characterized in that, include: Use camera equipment to acquire multi-angle physical images of power distribution panels; Extract matching feature points from the multi-angle entity image, and map the matching feature points into a three-dimensional sparse point cloud according to the spatial coordinates of the camera device; The three-dimensional sparse point cloud is densified to obtain a dense point cloud; Construct a static model of the power distribution panel based on the dense point cloud; Displaying the real-time acquired operating information of the power distribution panel in the static model of the power distribution panel includes: using a camera to capture real-time color images of the front of the power distribution panel in a timed sequence; using a Faster R-CNN deep learning network to identify and extract switches, terminals, text, and status lights from the color images of the front of the power distribution panel and locate and extract interest anchor boxes; for the text area image selected by the interest anchor box, using OCR technology to extract and recognize the text in the text area image, accurately obtaining the working status of the power distribution panel, and displaying the working status of the power distribution panel as the operating information of the power distribution panel in the static model of the power distribution panel; The step of extracting matching feature points from multi-angle entity images and mapping the matching feature points into a three-dimensional sparse point cloud based on the spatial coordinates of the camera device includes: Identify the power distribution panel area in the multi-angle entity image; The distribution panel area is segmented according to the GrabCut algorithm to obtain the foreground image of the power distribution panel; Extract dense feature points from the foreground image and perform feature point pair matching on the dense feature points to obtain matched feature point pairs; Remove mismatched feature points from the matching feature point pairs; Based on the spatial coordinates of the camera device, the matched feature points, after removing mismatched feature points, are mapped into a three-dimensional sparse point cloud; The construction of a static model of the power distribution panel based on the dense point cloud includes: The reprojection error of the dense point cloud is constrained by the speed of light adjustment method, and the surface reconstruction of the dense point cloud is performed by the Poisson surface reconstruction algorithm to obtain the static model of the power distribution panel. The process of extracting dense feature points from the foreground image and performing feature point pair matching on the dense feature points to obtain matched feature point pairs includes: The Harris corner detection algorithm is used to extract dense feature points in the foreground image; The KLT algorithm is used to track and match dense feature points to obtain matching feature point pairs; The process of removing mismatched feature points from the matched feature points includes: A preset number of sample feature points are selected from the matching feature points, wherein the preset number of sample feature points are not collinear; Construct a matrix model based on the preset number of sample feature points; Calculate the projection error between each of the matched feature points and the matrix model; If the projection error is greater than or equal to the preset error threshold, the matching feature point will be discarded as a mismatched feature point.

2. The power distribution panel display method based on computer vision according to claim 1, characterized in that, The step of densifying the three-dimensional sparse point cloud to obtain a dense point cloud includes: Obtain depth maps of multi-angle entity images at different angles; Based on the image neighborhood information obtained from the depth map, the 3D sparse point cloud is subjected to patch expansion and patch filtering to obtain the dense point cloud.

3. The power distribution panel display method based on computer vision according to claim 1, characterized in that, Before extracting dense feature points from the foreground image, the following steps are also included: The emphasize operator is used to enhance the image contrast of the foreground image.

4. The computer vision-based power distribution panel display method according to claim 1 further includes: The SURF algorithm is used to extract sparse feature points from the foreground image of the power distribution panel to obtain the spatial coordinates of the camera device.

5. A power distribution screen display device based on computer vision, characterized in that, include: The image acquisition unit is used to acquire multi-angle entity images of the power distribution panel; The feature point matching unit is used to extract matching feature points in the multi-angle entity image and map the matching feature points into a three-dimensional sparse point cloud according to the spatial coordinates of the camera device. The feature point processing unit is used to densify the three-dimensional sparse point cloud to obtain a dense point cloud. The model building unit is used to build a static model of the power distribution panel based on the dense point cloud. The information display unit is used to display the real-time acquired operation information of the power distribution panel in the static model of the power distribution panel. It includes: using a camera to capture and acquire color images of the front of the power distribution panel in real time at regular intervals; using a Faster R-CNN deep learning network to identify and extract switches, terminals, text, and status lights from the color images of the front of the power distribution panel and locate and extract interest anchor boxes; for the text area image selected by the interest anchor box, using OCR technology to extract and recognize the text in the text area image, accurately obtain the working status of the power distribution panel, and display the working status of the power distribution panel as the operation information of the power distribution panel in the static model of the power distribution panel. The step of extracting matching feature points from multi-angle entity images and mapping the matching feature points into a three-dimensional sparse point cloud based on the spatial coordinates of the camera device includes: Identify the power distribution panel area in the multi-angle entity image; The distribution panel area is segmented according to the GrabCut algorithm to obtain the foreground image of the power distribution panel; Extract dense feature points from the foreground image and perform feature point pair matching on the dense feature points to obtain matched feature point pairs; Remove mismatched feature points from the matching feature point pairs; Based on the spatial coordinates of the camera device, the matched feature points, after removing mismatched feature points, are mapped into a three-dimensional sparse point cloud; The construction of a static model of the power distribution panel based on the dense point cloud includes: The reprojection error of the dense point cloud is constrained by the speed of light adjustment method, and the surface reconstruction of the dense point cloud is performed by the Poisson surface reconstruction algorithm to obtain the static model of the power distribution panel. The process of extracting dense feature points from the foreground image and performing feature point pair matching on the dense feature points to obtain matched feature point pairs includes: The Harris corner detection algorithm is used to extract dense feature points in the foreground image; The KLT algorithm is used to track and match dense feature points to obtain matching feature point pairs; The process of removing mismatched feature points from the matched feature points includes: A preset number of sample feature points are selected from the matching feature points, wherein the preset number of sample feature points are not collinear; Construct a matrix model based on the preset number of sample feature points; Calculate the projection error between each of the matched feature points and the matrix model; If the projection error is greater than or equal to the preset error threshold, the matching feature point will be discarded as a mismatched feature point.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the computer vision-based power distribution screen display method as described in any one of claims 1 to 4.

Images