A method, device and system for comprehensive monitoring of overhead lines

Abstract

The present invention relates to the field of image processing technology, and in particular to a method, device and system for comprehensive monitoring of overhead lines. The method comprises: obtaining continuous multiple frames of monitoring images and environmental data of overhead lines; obtaining corresponding difference images according to each two adjacent frames of monitoring images; obtaining the degree of image change based on the position and number of pixel points with non-zero pixel values ​​in each difference image; obtaining the degree of environmental change of each difference image by combining the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the degree of image change; analyzing the aggregation of the difference images based on the degree of image change and the degree of environmental change to obtain reference consistency, and performing feature pyramid fusion on the corresponding monitoring images to obtain the feature image of the overhead line, and then monitoring the overhead line. The present invention improves the processing effect of the monitoring image of the overhead line and ensures the accuracy of the monitoring result of the overhead line.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device and system for comprehensive monitoring of overhead lines. Background Art

[0002] An intelligent comprehensive monitoring device for overhead lines integrates multiple functions, including warning lights, temperature and humidity sensors, microphones, speakers, etc., to jointly ensure the safe operation of distribution overhead lines around the clock, detect problems in time and take corresponding measures. The device is installed on the overhead lines of the distribution network to collect image data of line channels and environmental information around poles, and transmit it to the intelligent visualization management system platform through wireless communication and firewalls. The platform realizes visual monitoring of the channel environment, hidden danger distribution, equipment status and system operation status of overhead lines through real-time online monitoring of front-end visualization intelligent monitoring. The monitoring device adopts the image intelligent analysis module of front-end deep learning, which can automatically identify hidden dangers in the on-site images and automatically capture alarms and upload them to the platform when abnormalities occur. Operation and maintenance personnel can view the on-site warning situation by remotely logging into the platform, thereby effectively reducing the work pressure of patrol personnel. However, the existing image recognition-based overhead line external force monitoring algorithm has a low recognition accuracy rate, which affects the accuracy of the comprehensive monitoring results of overhead lines. Summary of the invention

[0003] In order to solve the problem that the monitoring results of the existing methods are of low accuracy when performing comprehensive monitoring on overhead lines, the purpose of the present invention is to provide a comprehensive monitoring method, device and system for overhead lines. The technical solutions adopted are as follows:

[0004] In a first aspect, the present invention provides a method for comprehensive monitoring of overhead lines, the method comprising the following steps:

[0005] Obtaining continuous multi-frame monitoring images of overhead lines and their corresponding environmental data;

[0006] Obtain a corresponding difference image based on each two adjacent frames of surveillance images; obtain the image change degree of each difference image based on the position distribution and quantity distribution of the pixel points with non-zero pixel values ​​in each difference image; obtain the environmental change degree of each difference image based on the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree;

[0007] The aggregation of the difference images is analyzed based on the degree of image change and the degree of environmental change to obtain the reference consistency of each difference image; the corresponding monitoring images are subjected to feature pyramid fusion using the reference consistency to obtain the feature image of the overhead line, and the overhead line is monitored using the feature image;

[0008] The aggregation of the difference images is analyzed based on the degree of comprehensive image change and the degree of environmental change to obtain the reference consistency of each difference image, including:

[0009] Mapping the image change degree and the environmental change degree of all difference images into a two-dimensional rectangular coordinate system to obtain multiple data points, wherein the abscissa of the two-dimensional rectangular coordinate system is the image change degree, and the ordinate is the environmental change degree;

[0010] Use density clustering algorithm to cluster all data points to obtain several clusters;

[0011] According to the shape distribution characteristics of the cluster where each difference image is located, the reference consistency of each difference image is obtained.

[0012] Preferably, the step of obtaining the image change degree of each difference image based on the position distribution and quantity distribution of the pixels having non-zero pixel values ​​in each difference image comprises:

[0013] Perform convex hull detection on the difference sample points in the candidate difference image to obtain the convex hull area;

[0014] The image change degree of the candidate difference image is obtained according to the normalized result of the number ratio of the difference sample points and the total number of pixel points in the convex hull area, wherein the number ratio of the difference sample points is the ratio between the number of difference sample points and the total number of pixel points in the convex hull area, and both the number ratio and the normalized result are positively correlated with the image change degree;

[0015] The candidate difference image is any difference image; the difference sample point is a pixel point in the difference image whose pixel value is not 0.

[0016] Preferably, the step of combining the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree to obtain the environmental change degree of each difference image includes:

[0017] Obtaining the mean of the environmental data of each class respectively, and the means of the environmental data of all classes constitute an environmental mean vector;

[0018] According to the difference between the environmental data vector corresponding to the candidate difference image and the environmental mean vector, and the environmental change degree of the candidate difference image, the environmental change degree of the candidate difference image is obtained; the environmental data vector corresponding to the candidate difference image is the environmental data vector of the latter frame of the monitoring image in the two adjacent frames of monitoring images corresponding to the candidate difference image; wherein the environmental data vector of the monitoring image is composed of all types of environmental data.

[0019] Preferably, obtaining the degree of environmental change of the candidate difference image according to the difference between the environmental data vector corresponding to the candidate difference image and the environmental mean vector and the degree of environmental change of the candidate difference image comprises:

[0020] The difference vector between the environment data vector corresponding to the candidate difference image and the environment mean vector is used as the environment difference vector corresponding to the candidate difference image;

[0021] The degree of environmental change of the candidate difference image is obtained according to the environmental difference vector and the degree of image change of the candidate difference image, the modulus of the environmental difference vector is positively correlated with the degree of environmental change, and the degree of image change is negatively correlated with the degree of environmental change.

[0022] Preferably, obtaining the reference consistency of each difference image according to the shape distribution characteristics of the cluster to which each difference image belongs includes:

[0023] Calculate the ratio of the distance between the data point corresponding to the candidate difference image and the center point of the cluster to which it belongs and the projection length of the cluster to which the data point corresponding to the candidate difference image belongs on the horizontal axis;

[0024] Obtain a cluster whose projection on the horizontal axis overlaps with the cluster where the data points corresponding to the candidate difference image are located, and record it as a reference cluster corresponding to the candidate difference image;

[0025] The reference consistency of the candidate difference image is obtained according to the average value of the projection lengths of all reference cluster clusters on the horizontal axis and the length ratio, the length ratio is positively correlated with the reference consistency, and the average value of the projection lengths of all reference cluster clusters on the horizontal axis is negatively correlated with the reference consistency.

[0026] Preferably, the step of performing feature pyramid fusion on the corresponding monitoring image using the reference consistency to obtain a feature image of the overhead line includes:

[0027] The reference consistency of the difference image is used as the stitching weight, and the monitoring image corresponding to the difference image is fused with feature pyramid to obtain the feature image of the overhead line;

[0028] The monitoring image corresponding to the difference image is the latter monitoring image of two adjacent monitoring images corresponding to the difference image.

[0029] Preferably, the step of obtaining a corresponding difference image based on each two adjacent surveillance image frames includes:

[0030] According to the difference in pixel values ​​of pixels at the same position in each two adjacent frames of monitoring images, a difference image corresponding to each two adjacent frames of monitoring images is obtained.

[0031] In a second aspect, the present invention provides an overhead line integrated monitoring device, comprising a memory and a processor, wherein the processor executes a computer program stored in the memory to implement a method for integrated monitoring of overhead lines.

[0032] In a third aspect, the present invention further provides an overhead line integrated monitoring system, the system comprising:

[0033] A data acquisition module, used to obtain continuous multi-frame monitoring images of overhead lines and their corresponding environmental data;

[0034] An analysis module is used to obtain a corresponding difference image based on each two adjacent frames of monitoring images; based on the position distribution and quantity distribution of the pixel points with non-zero pixel values ​​in each difference image, obtain the image change degree of each difference image; and based on the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree, obtain the environmental change degree of each difference image;

[0035] The feature fusion module is used to analyze the aggregation of the difference image based on the degree of image change and the degree of environmental change, and obtain the reference consistency of each difference image; the reference consistency is used to perform feature pyramid fusion on the corresponding monitoring image to obtain the feature image of the overhead line;

[0036] The aggregation of the difference images is analyzed based on the degree of comprehensive image change and the degree of environmental change to obtain the reference consistency of each difference image, including:

[0037] Mapping the image change degree and the environmental change degree of all difference images into a two-dimensional rectangular coordinate system to obtain multiple data points, wherein the abscissa of the two-dimensional rectangular coordinate system is the image change degree, and the ordinate is the environmental change degree;

[0038] Use density clustering algorithm to cluster all data points to obtain several clusters;

[0039] According to the shape distribution characteristics of the cluster where each difference image is located, the reference consistency of each difference image is obtained.

[0040] The present invention has at least the following beneficial effects:

[0041] The present invention first obtains a difference image between multiple consecutive frames of monitoring images of the overhead line collected, and evaluates the image change degree based on the position distribution and quantity distribution of pixels with non-zero pixel values ​​in each difference image; and considering that the installation position of the monitoring equipment causes the overhead line to usually occupy a small area in the image, the detail features of different monitoring targets captured through the receptive field are greatly affected by environmental factors. Although the monitoring targets are different, there are still certain correlation features in the processing process, especially under the same environmental conditions, the feature extraction of the monitoring target is more reference-oriented. Therefore, the difference between the environmental data corresponding to the difference image and the overall distribution of the environmental data and the image change degree are combined to analyze the environmental change degree, and the aggregation of the difference image is analyzed based on the comprehensive image change degree and the environmental change degree, and the reference consistency of each difference image is obtained. Then, the corresponding monitoring image is subjected to feature pyramid fusion using the reference consistency to obtain a clear feature image for completing the monitoring of the overhead line, thereby improving the processing effect of the overhead line monitoring image and ensuring the accuracy of the overhead line monitoring result. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a comprehensive monitoring method for overhead lines provided by an embodiment of the present invention;

[0044] Figure 2 The ILSM-SSD network structure diagram based on feature pyramid fusion provided by an embodiment of the present invention;

[0045] Figure 3 This is a structural block diagram of an overhead line integrated monitoring system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of an overhead line comprehensive monitoring method, device and system proposed in accordance with the present invention in conjunction with the accompanying drawings and preferred embodiments.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0048] The specific scheme of the overhead line comprehensive monitoring method, device and system provided by the present invention is described in detail below with reference to the accompanying drawings.

[0049] An overhead line comprehensive monitoring method embodiment:

[0050] The specific scenario targeted by this embodiment is: in the process of monitoring overhead lines, monitoring videos of overhead lines are collected by using equipment. The monitoring videos are composed of multiple frames of continuous monitoring images. Since the working environment of the equipment is relatively complex and the monitoring accuracy of the equipment depends on the detection accuracy of the acquired images for possible external force damage to the equipment and objects, the image feature extraction can be adjusted through the joint influence of multiple types of environmental monitoring information of the working environment of the equipment. This embodiment processes the monitoring images according to the changing characteristics of the pixels in the monitoring images and the changing characteristics of the environmental data, so as to improve the analysis rate and accuracy of the monitoring images of external force damage conditions in complex environments.

[0051] This embodiment proposes a comprehensive monitoring method for overhead lines, such as Figure 1 As shown, a comprehensive monitoring method for overhead lines in this embodiment includes the following steps:

[0052] Step S1, obtaining multiple frames of continuous monitoring images of the overhead line and their corresponding environmental data.

[0053] First, a front-end monitoring device is installed at a suitable location. The front-end monitoring device mainly includes a new monitoring device, a camera, a power supply unit and a sensor. The new monitoring device and the monitoring camera constitute a video image acquisition unit. The model obtained after training the external force damage front-end recognition algorithm based on feature fusion ILSM-SSD runs on the embedded hardware platform to realize image recognition at the edge. In addition to the video image acquisition unit, the front-end monitoring device is also connected to a variety of sensors such as micro-meteorological sensors and tower tilt sensors. Its main purpose is to consider other data such as overhead line channel environmental data and tower body data to help perceive the operating environment of overhead lines from multiple aspects. For example, temperature, wind speed, rainfall, etc. will have a certain impact on the aging speed of overhead lines. The tower tilt angle monitoring can reflect whether there is a possibility of collapse of the line tower. The environmental data collected in this embodiment include temperature, wind speed, and rainfall. In actual engineering applications, sensors can be configured according to actual needs.

[0054] In this embodiment, the time intervals for collecting monitoring images and environmental data are set to 1 second. Continuous multiple-frame monitoring images of the overhead lines and their corresponding environmental data within the current time period are collected, and the collected monitoring images and environmental data are uploaded to the platform. The current time period is a set of all historical moments whose time intervals with the current moment are less than or equal to the preset time length. In this embodiment, the preset time length is 10 minutes. In specific applications, the implementer can set it according to the specific situation.

[0055] So far, this embodiment has collected multiple frames of continuous monitoring images of the overhead line in the current time period and their corresponding environmental data.

[0056] Step S2, obtaining a corresponding difference image based on each two adjacent frames of monitoring images; obtaining the image change degree of each difference image based on the position distribution and quantity distribution of pixel points with non-zero pixel values ​​in each difference image; and obtaining the environmental change degree of each difference image by combining the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree.

[0057] Since the overhead line external force damage monitoring equipment is installed at a high position and requires a wide monitoring range, the target occupies a small area in the monitoring image. In addition, the overhead line scene is complex and has many interferences, which increases the difficulty of identification. When the environmental factors change, the clarity of the picture and the change of the target range caused by the shooting angle of the device are more affected. The receptive field in the convolution process based on the original lightweight network MobileNet-SSD depends on the understanding ability of the target area for information extraction in the image. When the target occupies a small proportion in the picture, the feature extraction of the image is more affected by environmental factors. Accordingly, the matching of the fuzzy state changes of the image affected by environmental factors can better adapt to the overall image feature changes of the image obtained by the overhead line external force damage monitoring equipment, which is convenient for subsequent comparative analysis and extraction of target features. Therefore, it is necessary to analyze the impact of environmental factors, realize the reference extraction of image targets under different environmental factors, and then analyze the extracted monitoring images.

[0058] First, according to the difference in pixel values ​​of the pixels at the same position in each of the two adjacent frames of surveillance images, a difference image corresponding to each of the two adjacent frames of surveillance images is obtained. The method for obtaining the difference image is a prior art and will not be described in detail here. It should be noted that all surveillance images collected in this embodiment are of equal size.

[0059] Since the pixel positions with non-zero pixel values ​​in the difference image represent the target positions that have changed in the image, the larger the distribution range, the more it indicates that the image change is due to the overall difference in the image caused by changes in environmental factors. The smaller the distribution range, the more likely it is that the image area of ​​the monitored target has changed due to the movement of the monitored target or the vibration offset of the monitoring equipment caused by environmental factors.

[0060] Next, any difference image is taken as an example for description, and other difference images can be processed using the method provided in this embodiment.

[0061] Specifically, any difference image is recorded as a candidate difference image, and the pixels in the difference image with non-zero pixel values ​​are recorded as difference sample points. The difference sample points in the candidate difference image are subjected to convex hull detection to obtain the convex hull area. The convex hull detection is a prior art and will not be described in detail here. The ratio between the number of difference sample points and the total number of pixels in the convex hull area is taken as the number ratio of the difference sample points; the image change degree of the candidate difference image is obtained based on the normalized result of the number ratio of the difference sample points and the total number of pixels in the convex hull area, and the number ratio and the normalized result are both positively correlated with the image change degree.

[0062] Among them, the positive correlation means that the dependent variable will increase as the independent variable increases, and the dependent variable will decrease as the independent variable decreases. It can be an additive relationship, a multiplicative relationship, etc., which is determined by practical applications.

[0063] In this embodiment, a specific calculation formula for the image change degree is given, and the image change degree of the i-th difference image can be expressed as:

[0064]

[0065] Among them, A i Represented as the image change degree of the i-th difference image, N i represents the number of difference sample points in the i-th difference image, S i It represents the total number of pixels in the convex hull area of ​​the i-th difference image, and norm() represents the linear normalization function.

[0066] represents the ratio of the number of difference sample points in the ith difference image, which is used to reflect the relative distribution density of the difference sample points, that is, the target variation range of adjacent monitoring images of overhead lines; norm(S i ) represents the normalized result of the total number of pixels in the convex hull area, which is used as the weight of the proportion of the number of difference sample points. The more the total number of pixels in the convex hull area, the greater the corresponding image change.

[0067] Due to the installation location of the monitoring equipment, the monitoring target usually occupies a small area in the image. For the neural network, the detailed features captured by the receptive field of different monitoring targets, such as different large-scale mechanical equipment on the construction site: cranes, bulldozers, etc., are actually greatly affected by environmental factors. Correspondingly, for the image, although the monitoring targets are different, there are still certain related features in the processing process, especially under the same environmental conditions, which is more relevant for the feature extraction of the monitoring target.

[0068] The mean of each type of environmental data collected in the current time period is obtained respectively. It should be noted that: each type of environmental data corresponds to a mean; the means of all types of environmental data constitute an environmental mean vector, the first element in the environmental mean vector is the mean of the first type of environmental data, the second element in the environmental mean vector is the mean of the second type of environmental data, the third element in the environmental mean vector is the mean of the third type of environmental data, and so on. The last element in the environmental mean vector is the mean of the last type of environmental data.

[0069] For any frame of surveillance image, all types of environmental data collected when acquiring the frame of surveillance image constitute the environmental data vector of the frame of surveillance image. The first element in the environmental data vector of the frame of surveillance image is the first type of environmental data collected when acquiring the frame of surveillance image, the second element in the environmental mean vector is the second type of environmental data collected when acquiring the frame of surveillance image, the third element in the environmental mean vector is the third type of environmental data collected when acquiring the frame of surveillance image, and so on. The last element in the environmental mean vector is the last type of environmental data collected when acquiring the frame of surveillance image.

[0070] Next, this embodiment is still explained by taking any candidate difference image as an example. Specifically, the difference vector between the environment data vector corresponding to the candidate difference image and the environment mean vector is used as the environment difference vector corresponding to the candidate difference image; the environment change degree of the candidate difference image is obtained according to the environment difference vector corresponding to the candidate difference image and the environment change degree of the candidate difference image; the environment data vector corresponding to the candidate difference image is the environment data vector of the latter frame of the monitoring image between the two adjacent frames of monitoring images corresponding to the candidate difference image.

[0071] The degree of environmental change of the candidate difference image is obtained according to the environmental difference vector and the degree of image change of the candidate difference image, the modulus of the environmental difference vector is positively correlated with the degree of environmental change, and the degree of image change is negatively correlated with the degree of environmental change.

[0072] Among them, a positive correlation means that the dependent variable will increase as the independent variable increases, and the dependent variable will decrease as the independent variable decreases. It can be an additive relationship, a multiplicative relationship, etc., which is determined by actual application; a negative correlation means that the dependent variable will decrease as the independent variable increases, and the dependent variable will increase as the independent variable decreases. It can be a subtractive relationship, a division relationship, etc., which is determined by actual application.

[0073] In this embodiment, a specific calculation formula for the degree of environmental change is given, and the degree of environmental change of the i-th difference image can be expressed as:

[0074]

[0075] Among them, B i Indicates the degree of environmental change of the i-th difference image, A i Indicates the degree of image change of the i-th difference image, H i represents the environmental data vector corresponding to the i-th difference image, represents the environmental mean vector, ‖‖ represents the modulus length, and norm() represents the linear normalization function.

[0076] represents the environmental difference vector corresponding to the i-th difference image, The modulus length of the environment difference vector corresponding to the ith difference image is used to reflect the overall difference between the environment data corresponding to the ith difference image and all the environment data. The larger the modulus length, the greater the overall difference between the two. It reflects that when the difference image has a similar degree of image change, the greater the change of its environmental data vector, the more universal the change of the difference image is, and the smaller the impact on the detail capture ability of the receptive field in the convolution process of the neural network, and the greater the reliability of the feature extraction of analog multi-environmental image. When the modulus length of the environmental difference vector corresponding to the i-th difference image is longer and the image change degree of the i-th difference image is smaller, the degree of environmental change of the i-th difference image is greater.

[0077] By adopting the above method, the image change degree and the environmental change degree of each difference image can be obtained.

[0078] Step S3, analyzing the aggregation of the difference images based on the degree of image change and the degree of environmental change to obtain the reference consistency of each difference image; using the reference consistency to perform feature pyramid fusion on the corresponding monitoring image to obtain the feature image of the overhead line, and using the feature image to monitor the overhead line.

[0079] The degree of environmental change is analyzed more from the perspective of changes in a single difference image. In fact, the distribution of overall environmental data changes and image change types should be used to more accurately measure the comparable reference between images.

[0080] The image change degree and environmental change degree of all difference images are mapped to a two-dimensional rectangular coordinate system to obtain multiple data points, wherein the horizontal coordinate of the two-dimensional rectangular coordinate system is the image change degree and the vertical coordinate is the environmental change degree; a density clustering algorithm is used to cluster all data points to obtain multiple cluster clusters. The density clustering algorithm is a prior art and will not be described in detail here. The higher the degree of clustering of data points in the cluster, the higher its reference consistency. Therefore, this embodiment determines the reference consistency of each difference image by clustering the difference images according to the degree of image change and the degree of environmental change.

[0081] This embodiment is still described by taking the candidate difference image as an example. Specifically, the ratio of the distance between the data point corresponding to the candidate difference image and the center point of the cluster to which it belongs and the length of the projection of the cluster to which the data point corresponding to the candidate difference image belongs on the horizontal axis is calculated; the clusters with overlapping projections on the horizontal axis with the cluster to which the data point corresponding to the candidate difference image belongs are obtained, and recorded as the reference clusters corresponding to the candidate difference image. The reference consistency of the candidate difference image is obtained according to the average value of the projection lengths of all reference clusters on the horizontal axis and the length ratio. The length ratio is positively correlated with the reference consistency, and the average value of the projection lengths of all reference clusters on the horizontal axis is negatively correlated with the reference consistency.

[0082] Among them, a positive correlation means that the dependent variable will increase as the independent variable increases, and the dependent variable will decrease as the independent variable decreases. It can be an additive relationship, a multiplicative relationship, etc., which is determined by actual application; a negative correlation means that the dependent variable will decrease as the independent variable increases, and the dependent variable will increase as the independent variable decreases. It can be a subtractive relationship, a division relationship, etc., which is determined by actual application.

[0083] In this embodiment, a specific calculation formula for reference consistency is given, and the reference consistency of the i-th difference image can be expressed as:

[0084]

[0085] Among them, C i represents the reference consistency of the i-th difference image, l i Indicates the distance between the data point corresponding to the i-th difference image and the center point of its cluster, L i represents the projection length of the cluster where the data point corresponding to the ith difference image is located on the horizontal axis, K represents the number of reference clusters corresponding to the ith difference image, and Li,k It represents the projection length of the kth reference cluster corresponding to the i-th difference image on the horizontal axis, and norm() represents the linear normalization function.

[0086] l i / L i It represents the ratio of the distance between the data point corresponding to the ith difference image and the center point of the cluster to which it belongs and the projection length of the cluster to which the data point corresponding to the ith difference image belongs on the horizontal axis. Represents the average projection length of all reference clusters corresponding to the ith difference image on the horizontal axis. The greater the ratio between the distance between the data point corresponding to the ith difference image and the center point of the cluster to which it belongs and the projection length of the cluster to which the data point corresponding to the ith difference image belongs on the horizontal axis, and the smaller the average projection length of all reference clusters corresponding to the ith difference image on the horizontal axis, the higher the reference consistency of the ith difference image.

[0087] By adopting the above method, the reference consistency of each difference image can be obtained.

[0088] Generally, feature fusion can be achieved by concatenation or element-wise sum. The feature map sizes of the six feature maps used for target detection are different. The element-wise sum method requires the feature size and the number of channels to be consistent. Therefore, this embodiment uses concatenation to perform feature pyramid fusion from bottom to top. Based on the above ideas, the ILSM-SSD network structure based on feature pyramid fusion is constructed as follows: Figure 2 As shown in the figure, Non-Maximum Suppression represents non-maximum suppression, which is a commonly used post-processing step for removing redundant overlapping frames and retaining only the most likely detection frame; Detection represents detection classification, which is used for target detection classification and bounding box positioning, that is, determining the type of target and the position of the object; the numbers 38, 19, 10, 5, 3, and 1 in the figure are used to represent the size of the feature size selected during multi-scale feature detection. In this embodiment, six feature maps of 38*38, 19*19, 10*10, 5*5, 3*3, and 1*1 are selected; Bneck 4_3 represents the anti-residual module combined with the linear bottleneck processing layer 1, Bneck6_3 represents the anti-residual module combined with the linear bottleneck processing layer, which reduces the amount of calculation and the amount of parameters; Conv8 represents the depth-separable convolution processing layer 1, Conv9_2 represents the depth-separable convolution processing layer 2, Conv10_2 represents the depth-separable convolution processing layer 3, and Conv11_2 represents the depth-separable convolution processing layer 4. The depth-separable convolution processing layer greatly reduces the network calculation amount and the complexity of the network structure while maintaining the detection accuracy.

[0089] The deep features are enlarged, and a feature map of the same size as the previous layer is obtained by double upsampling. The previous layer features are adjusted and then spliced ​​with the deep features. Finally, a 3*3 convolution is performed and the images are sent to the classifier for detection. This embodiment uses the reference consistency of the image to perform concat splicing of different degrees of the image set in the cluster. Specifically, the reference consistency of the difference image is used as the splicing weight, and the monitoring image corresponding to the difference image is subjected to feature pyramid fusion to obtain the feature image of the overhead line; wherein the monitoring image corresponding to the difference image is the latter frame of the monitoring image in the two adjacent frames of monitoring images corresponding to the difference image. The multi-source feature splicing process can be expressed as:

[0090] T concat =concat([T i ×C i ],axis)

[0091] Among them, T concat represents the surveillance image corresponding to the ith difference image, C i represents the reference consistency of the i-th difference image, T i It represents the tensor of the surveillance image corresponding to the i-th difference image after the concat operation, and axis represents the dimension of the splicing.

[0092] Therefore, after continuous upsampling by the deep network conv11_2 and feature fusion with the upper layer, the semantic information in the feature image of the overhead line is gradually enriched and clear, and the details, texture, contour information, etc. contained in the feature image can be obtained, thereby monitoring the overhead line.

[0093] This embodiment first obtains the difference image between the collected continuous multiple frames of monitoring images of the overhead line, and evaluates the image change degree based on the position distribution and quantity distribution of the pixel points with non-zero pixel values ​​in each difference image; considering that the installation position of the monitoring equipment causes the overhead line to usually account for a small area in the image, the detail features of different monitoring targets captured through the receptive field are greatly affected by environmental factors. Although the monitoring targets are different, there are still certain related features in the processing process, especially under the same environmental conditions, the feature extraction of the monitoring target is more reference-oriented. Therefore, the degree of environmental change is analyzed by combining the difference between the environmental data corresponding to the difference image and the overall distribution of the environmental data and the degree of image change, and the aggregation of the difference image is analyzed based on the comprehensive image change degree and the degree of environmental change, and the reference consistency of each difference image is obtained, and then the corresponding monitoring image is subjected to feature pyramid fusion using the reference consistency to obtain a clear feature image for completing the monitoring of the overhead line, thereby improving the processing effect of the overhead line monitoring image and ensuring the accuracy of the overhead line monitoring result.

[0094] An embodiment of an overhead line integrated monitoring device:

[0095] An overhead line integrated monitoring device of this embodiment includes a memory and a processor, and the processor executes a computer program stored in the memory to implement the above-mentioned overhead line integrated monitoring method.

[0096] An overhead line integrated monitoring system embodiment:

[0097] like Figure 3 As shown, the figure shows a structural block diagram of an overhead line integrated monitoring system, the system is used to implement an overhead line integrated monitoring method, the system includes a data acquisition module, an analysis module and a feature fusion module;

[0098] The data acquisition module is used to obtain continuous multi-frame monitoring images of the overhead line and the corresponding environmental data;

[0099] An analysis module is used to obtain a corresponding difference image based on each two adjacent frames of monitoring images; based on the position distribution and quantity distribution of the pixel points with non-zero pixel values ​​in each difference image, obtain the image change degree of each difference image; and based on the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree, obtain the environmental change degree of each difference image;

[0100] The feature fusion module is used to analyze the aggregation of the difference image by comprehensively considering the degree of image change and the degree of environmental change, and obtain the reference consistency of each difference image; perform feature pyramid fusion on the corresponding monitoring image by using the reference consistency to obtain the feature image of the overhead line, and use the feature image to monitor the overhead line;

[0101] The aggregation of the difference images is analyzed based on the degree of comprehensive image change and the degree of environmental change to obtain the reference consistency of each difference image, including:

[0102] Mapping the image change degree and the environmental change degree of all difference images into a two-dimensional rectangular coordinate system to obtain multiple data points, wherein the abscissa of the two-dimensional rectangular coordinate system is the image change degree, and the ordinate is the environmental change degree;

[0103] Use density clustering algorithm to cluster all data points to obtain several clusters;

[0104] According to the shape distribution characteristics of the cluster where each difference image is located, the reference consistency of each difference image is obtained.

[0105] It should be understood that Figure 3The structural block diagram of an overhead line integrated monitoring system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art can understand that the above-mentioned method and system can be implemented using computer executable instructions and / or included in a processor control code, for example, such codes are provided on a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of this specification can not only be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software (for example, firmware).

[0106] For more details about the above modules, please refer to other places in this manual and will not be repeated here.

[0107] In other embodiments, a medium is also provided, wherein the medium stores at least one program executable by a computer, and when the at least one program is executed by a computer, the computer executes the steps in the overhead line comprehensive monitoring method in the above embodiment, and the medium may be a computer-readable storage medium.

[0108] Among them, the provided devices, systems, and media are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0109] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for comprehensive monitoring of overhead lines, characterized in that: The method comprises the following steps: Obtaining continuous multi-frame monitoring images of overhead lines and their corresponding environmental data; Obtain a corresponding difference image based on each two adjacent frames of surveillance images; obtain the image change degree of each difference image based on the position distribution and quantity distribution of the pixel points with non-zero pixel values ​​in each difference image; obtain the environmental change degree of each difference image based on the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree; The aggregation of the difference images is analyzed based on the degree of image change and the degree of environmental change to obtain the reference consistency of each difference image; the corresponding monitoring images are subjected to feature pyramid fusion using the reference consistency to obtain the feature image of the overhead line, and the overhead line is monitored using the feature image; The aggregation of the difference images is analyzed based on the degree of comprehensive image change and the degree of environmental change to obtain the reference consistency of each difference image, including: Mapping the image change degree and the environmental change degree of all difference images into a two-dimensional rectangular coordinate system to obtain multiple data points, wherein the abscissa of the two-dimensional rectangular coordinate system is the image change degree, and the ordinate is the environmental change degree; Use density clustering algorithm to cluster all data points to obtain several clusters; According to the shape distribution characteristics of the cluster where each difference image is located, the reference consistency of each difference image is obtained.

2. A method for comprehensive monitoring of overhead lines according to claim 1, characterized in that: The step of obtaining the image change degree of each difference image based on the position distribution and quantity distribution of the pixels having non-zero pixel values ​​in each difference image comprises: Perform convex hull detection on the difference sample points in the candidate difference image to obtain the convex hull area; The image change degree of the candidate difference image is obtained according to the normalized result of the number ratio of the difference sample points and the total number of pixel points in the convex hull area, wherein the number ratio of the difference sample points is the ratio between the number of difference sample points and the total number of pixel points in the convex hull area, and both the number ratio and the normalized result are positively correlated with the image change degree; The candidate difference image is any difference image; the difference sample point is a pixel point in the difference image whose pixel value is not 0.

3. A method for comprehensive monitoring of overhead lines according to claim 2, characterized in that: The step of combining the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree to obtain the environmental change degree of each difference image includes: Obtaining the mean of the environmental data of each class respectively, and the means of the environmental data of all classes constitute an environmental mean vector; According to the difference between the environmental data vector corresponding to the candidate difference image and the environmental mean vector, and the environmental change degree of the candidate difference image, the environmental change degree of the candidate difference image is obtained; the environmental data vector corresponding to the candidate difference image is the environmental data vector of the latter frame of the monitoring image in the two adjacent frames of monitoring images corresponding to the candidate difference image; wherein the environmental data vector of the monitoring image is composed of all types of environmental data.

4. A method for comprehensive monitoring of overhead lines according to claim 3, characterized in that: The step of obtaining the degree of environmental change of the candidate difference image according to the difference between the environmental data vector corresponding to the candidate difference image and the environmental mean vector and the degree of environmental change of the candidate difference image comprises: The difference vector between the environment data vector corresponding to the candidate difference image and the environment mean vector is used as the environment difference vector corresponding to the candidate difference image; The degree of environmental change of the candidate difference image is obtained according to the environmental difference vector and the degree of image change of the candidate difference image, the modulus of the environmental difference vector is positively correlated with the degree of environmental change, and the degree of image change is negatively correlated with the degree of environmental change.

5. The method for comprehensive monitoring of overhead lines according to claim 2, characterized in that: The step of obtaining the reference consistency of each difference image according to the shape distribution characteristics of the cluster in which each difference image is located includes: Calculate the ratio of the distance between the data point corresponding to the candidate difference image and the center point of the cluster to which it belongs and the projection length of the cluster to which the data point corresponding to the candidate difference image belongs on the horizontal axis; Obtain a cluster whose projection on the horizontal axis overlaps with the cluster where the data points corresponding to the candidate difference image are located, and record it as a reference cluster corresponding to the candidate difference image; The reference consistency of the candidate difference image is obtained according to the average value of the projection lengths of all reference cluster clusters on the horizontal axis and the length ratio, the length ratio is positively correlated with the reference consistency, and the average value of the projection lengths of all reference cluster clusters on the horizontal axis is negatively correlated with the reference consistency.

6. The method for comprehensive monitoring of overhead lines according to claim 1, characterized in that: The step of performing feature pyramid fusion on the corresponding monitoring image by using the reference consistency to obtain a feature image of the overhead line includes: The reference consistency of the difference image is used as the stitching weight, and the monitoring image corresponding to the difference image is fused with feature pyramid to obtain the feature image of the overhead line; The monitoring image corresponding to the difference image is the latter monitoring image of two adjacent monitoring images corresponding to the difference image.

7. The method for comprehensive monitoring of overhead lines according to claim 1, characterized in that: The step of obtaining a corresponding difference image according to each two adjacent surveillance image frames includes: According to the difference in pixel values ​​of pixels at the same position in each two adjacent frames of monitoring images, a difference image corresponding to each two adjacent frames of monitoring images is obtained.

8. An overhead line integrated monitoring device, comprising a memory and a processor, characterized in that: The processor executes the computer program stored in the memory to implement the overhead line comprehensive monitoring method according to any one of claims 1 to 7.

9. An overhead line integrated monitoring system, characterized in that: The system is used to implement the overhead line comprehensive monitoring method according to claim 1, and the system comprises: A data acquisition module, used to obtain continuous multi-frame monitoring images of overhead lines and their corresponding environmental data; An analysis module is used to obtain a corresponding difference image based on each two adjacent frames of monitoring images; based on the position distribution and quantity distribution of the pixel points with non-zero pixel values ​​in each difference image, obtain the image change degree of each difference image; and based on the difference between the environmental data corresponding to each difference image and the overall distribution of the environmental data and the image change degree, obtain the environmental change degree of each difference image; The feature fusion module is used to analyze the aggregation of the difference image based on the degree of image change and the degree of environmental change, and obtain the reference consistency of each difference image; the reference consistency is used to perform feature pyramid fusion on the corresponding monitoring image to obtain the feature image of the overhead line; The aggregation of the difference images is analyzed based on the degree of comprehensive image change and the degree of environmental change to obtain the reference consistency of each difference image, including: Mapping the image change degree and the environmental change degree of all difference images into a two-dimensional rectangular coordinate system to obtain multiple data points, wherein the abscissa of the two-dimensional rectangular coordinate system is the image change degree, and the ordinate is the environmental change degree; Use density clustering algorithm to cluster all data points to obtain several clusters; According to the shape distribution characteristics of the cluster where each difference image is located, the reference consistency of each difference image is obtained.

Images