Power plant early warning detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a power plant early warning detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring image data of a target power plant area; the image data comprises multispectral images at different visual angles; carrying out image splicing processing on the multispectral images at different visual angles to obtain first image data; performing image fusion processing on the first image data based on a preset image fusion technology to obtain second image data; performing target detection and identification on the second image data by using a deep learning technology to obtain an anomaly detection result of the target power plant area; and performing early warning reminding processing in response to an abnormal detection result. The embodiment of the invention can be applied to regular inspection and fault diagnosis of power plant equipment, monitoring and evaluation of power plant environmental influence, power plant safety management and risk control, optimization of power plant operation efficiency, cost control and the like based on the acquired image data of the power plant, and can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a power plant early warning detection method, device, electronic equipment and storage medium. Background Art

[0002] As the core facility of power production, the safe and stable operation of power plants is of vital importance. However, during the operation of power plant equipment, due to various reasons such as long-term high-load operation, environmental factors and equipment aging, it may lead to safety hazards such as equipment temperature rise and fire. If these hidden dangers are not discovered and handled in time, they will seriously threaten the safe operation of power plants and even cause major accidents.

[0003] At present, the safety monitoring and early warning of power plants mainly rely on manual inspections, which have problems such as long inspection cycles, low efficiency, and easy missed inspections, and cannot achieve real-time and comprehensive monitoring of power plant equipment. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to propose a power plant early warning detection method, device, electronic equipment and storage medium, in order to solve at least one problem of the prior art. The present invention can efficiently and conveniently realize power plant early warning detection.

[0005] To achieve the above object, an embodiment of the present invention provides a power plant early warning detection method, the method comprising:

[0006] Acquire image data of the target power plant area; the image data includes multispectral images at different viewing angles;

[0007] Performing image stitching processing on multispectral images with different viewing angles to obtain first image data;

[0008] Performing image fusion processing on the first image data based on a preset image fusion technology to obtain second image data;

[0009] Using deep learning technology to perform target detection and recognition on the second image data, and obtain anomaly detection results of the target power plant area;

[0010] Perform early warning and reminder processing in response to abnormal detection results.

[0011] In some embodiments, acquiring image data of a target power plant area includes the following steps:

[0012] Obtain image data collected by the target UAV during its flight over the target power plant area;

[0013] Among them, the target UAV is equipped with a multispectral sensor; the image data is collected based on the multispectral sensor.

[0014] In some embodiments, before the step of performing image stitching processing on the multispectral images at different viewing angles, the method further includes the following steps:

[0015] Preprocess the image data;

[0016] The image stitching process is performed based on the pre-processed image data.

[0017] In some embodiments, the image data is acquired based on a preset multispectral sensor; preprocessing the image data includes at least one of the following steps:

[0018] Perform radiometric correction on image data based on radiometric calibration parameters of multispectral sensors;

[0019] Perform atmospheric correction on image data based on preset radiation transfer model simulation;

[0020] The image data is geometrically corrected using polynomial deformation techniques and bilinear interpolation.

[0021] In some embodiments, performing image stitching processing on multispectral images of different viewing angles to obtain first image data includes the following steps:

[0022] The visual geometry group network is used to independently encode multispectral images of different perspectives, and the feature pyramid corresponding to the multispectral image of each perspective is obtained;

[0023] Among them, the feature pyramid is obtained by connecting the image features obtained at each level of the visual geometry group network;

[0024] Based on the feature pyramid, spatial graph reasoning is performed through a graph convolutional network to obtain the first image data.

[0025] In some embodiments, the feature pyramid includes a first feature pyramid of a reference viewpoint and a second feature pyramid of a target viewpoint; based on the feature pyramid, spatial graph reasoning is performed through a graph convolution network to obtain first image data, including the following steps:

[0026] Perform multi-scale feature extraction on the first feature pyramid to obtain a first node representation;

[0027] The multi-scale feature extraction includes a pooling operation, a convolution operation, and an upsampling operation in sequence; the first node represents an extracted feature corresponding to a feature of each scale in the first feature pyramid;

[0028] Perform multi-scale feature extraction on the second feature pyramid to obtain a second node representation;

[0029] The second node represents the extracted features corresponding to the features of each scale in the second feature pyramid;

[0030] A progressive mechanism is used to perform associative reasoning on the extracted features represented by the first node and the extracted features represented by the second node, so as to obtain the guide nodes of each scale corresponding to the reference viewpoint and the target viewpoint;

[0031] Based on the guide node, the homography parameters of the reference viewpoint and the target viewpoint are obtained by processing the connection layer and the fully connected layer;

[0032] Based on the homography parameters, the view positions corresponding to the target viewpoint and the reference viewpoint are converted to the same plane, and then the first image data is obtained by using the reconstruction and calculation loss function based on the space and channel map.

[0033] In some embodiments, performing image fusion processing on the first image data based on a preset image fusion technology to obtain the second image data includes the following steps:

[0034] A region-based fusion method is used to perform region enhancement processing on the first image data to obtain second image data.

[0035] In some embodiments, using deep learning technology to perform target detection and recognition on the second image data to obtain an abnormality detection result of the target power plant area includes the following steps:

[0036] Using the pre-trained target detection model to perform target detection and recognition on the second image data, and obtain an abnormality detection result of the target power plant area;

[0037] Among them, the target detection model is trained based on the fused image data with anomaly detection labels.

[0038] To achieve the above object, another aspect of an embodiment of the present invention provides a power plant early warning detection device, the device comprising:

[0039] The first module is used to obtain image data of the target power plant area; the image data includes multispectral images at different viewing angles;.

[0040] The second module is used to perform image stitching processing on multispectral images with different viewing angles to obtain first image data;

[0041] A third module is used to perform image fusion processing on the first image data based on a preset image fusion technology to obtain second image data;

[0042] The fourth module is used to perform target detection and recognition on the second image data using deep learning technology to obtain an abnormality detection result of the target power plant area;

[0043] The fifth module is used to perform early warning and reminder processing in response to abnormal detection results.

[0044] In some embodiments, the apparatus further comprises:

[0045] The sixth module is used for preprocessing the image data;

[0046] The image stitching process is performed based on the pre-processed image data.

[0047] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0048] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0049] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a power plant early warning detection method, device, electronic device and storage medium, the scheme acquires image data of the target power plant area; the image data includes multispectral images at different viewing angles; the multispectral images at different viewing angles are subjected to image stitching processing to obtain first image data; the first image data is subjected to image fusion processing based on a preset image fusion technology to obtain second image data; the second image data is subjected to target detection and recognition using deep learning technology to obtain abnormal detection results of the target power plant area; and early warning reminder processing is performed in response to the abnormal detection results. The embodiments of the present invention can be applied to regular inspection and fault diagnosis of power plant equipment, monitoring and evaluation of environmental impacts of power plants, safety management and risk control of power plants, optimization of power plant operating efficiency and cost control, etc. based on the acquired image data of power plants. The present invention can efficiently and conveniently realize early warning detection of power plants. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a power plant early warning detection method provided by an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the principle of linear difference provided by an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of the principle of bilinear interpolation provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the hierarchical structure of a graph convolutional network provided by an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of the overall process of the power plant early warning detection method provided by an embodiment of the present invention;

[0055] Figure 6It is a schematic diagram of a program flow of a power plant early warning detection method provided by an embodiment of the present invention;

[0056] Figure 7 It is a structural schematic diagram of a power plant early warning detection device provided by an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0059] It is understood that the terms "first", "second", etc. used in the present invention can be used to describe various concepts in the present invention, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0060] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0062] The power plant early warning detection method provided by the embodiment of the present invention relates to the field of image processing technology. The power plant early warning detection method provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the power plant early warning detection method, etc., but is not limited to the above forms.

[0063] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] Figure 1 is an optional flow chart of a power plant early warning detection method provided by an embodiment of the present invention. Figure 1 The method may include but is not limited to steps S100 to S500.

[0065] S100, acquiring image data of a target power plant area;

[0066] Among them, the image data includes multispectral images at different viewing angles;

[0067] It should be noted that, in some embodiments, obtaining image data of the target power plant area may include the following steps: obtaining image data collected by the target UAV during the flight of the target power plant area; wherein the target UAV is provided with a multispectral sensor; and the image data is collected based on the multispectral sensor.

[0068] For example, in some specific implementations, data collection is first performed: a drone equipped with a multispectral sensor is used to take aerial photos of the power plant area to obtain image data in different bands. These data may include images in different spectral ranges such as visible light and near infrared, as well as images at different viewing angles, to ensure full coverage of the power plant area.

[0069] In some embodiments, before executing step S200, the method may further include the following steps: preprocessing the image data; wherein the image stitching processing is performed based on the preprocessed image data.

[0070] For example, in some specific implementations, the collected images may be preprocessed: radiation correction and atmospheric correction are performed on the collected multispectral images to eliminate the influence of illumination and atmospheric conditions on the image quality. In addition, geometric correction of the images is also required to ensure geometric consistency between images.

[0071] Further, in some optional implementations, the image data is acquired based on a preset multispectral sensor; and preprocessing the image data may include at least one of the following steps:

[0072] Perform radiometric correction on image data based on radiometric calibration parameters of multispectral sensors;

[0073] For example, in some embodiments, during the image preprocessing stage, the purpose of radiometric correction is to eliminate the non-uniformity of the sensor itself and convert digital counts (DN) into radiometric brightness values. This usually involves using the sensor's radiometric calibration parameters, such as gain and offset, and considering factors such as the sun angle and the earth-sun distance.

[0074] Perform atmospheric correction on image data based on preset radiation transfer model simulation;

[0075] Exemplarily, in some specific embodiments, atmospheric correction aims to eliminate the influence of the atmosphere on the image, including scattering and absorption. This can be done by using a radiation transfer model, such as the 6S model, to simulate the influence of the atmosphere on the spectral signal and establish an atmospheric correction coefficient lookup table. Due to the inherent characteristics of the imaging optical system, as well as the inclination angle and rotation angle between the lens imaging plane and the object plane, the acquired image has nonlinear geometric distortion. These distortions must be eliminated before quantitative analysis and detection of the image.

[0076] The image data is geometrically corrected using polynomial deformation techniques and bilinear interpolation.

[0077] For example, in some specific implementations, in the geometric correction stage, generally speaking, for wide-angle lens imaging, as long as (cubic) polynomial deformation technology and bilinear interpolation are used, this type of distortion can be eliminated to obtain an ideal image. Among them, the use of polynomial fitting for spatial coordinate transformation is a key step. In geometric distortion image correction, a reference image is often used to correct the distorted image. Assuming the coordinates of the reference image are (u, v) and the coordinates of the distorted image are (x, y), the two-dimensional polynomial mathematical model used in the resampling mapping method is:

[0078]

[0079] (1) Where n is the number of terms in the polynomial, and aij and bij are the coefficients of each term. Let the reference image be g(u, v) and the distorted image be f(x, y). For the same point in the image, assume that its grayscale value is constant, that is:

[0080] g(u,v)=f(x,y) (2)

[0081] Formula (2) is the basic formula for image geometric distortion correction. X and Y in the reference image are integers, but X and Y calculated by formula (1) are not necessarily integers, and grayscale interpolation operations must be performed.

[0082] Grayscale interpolation methods are commonly used in the nearest neighbor method, bilinear interpolation and cubic convolution. Theoretically, the nearest neighbor method has the worst effect, the cubic convolution method has the best effect, and the bilinear interpolation method is between the two. The cubic convolution method has high calculation accuracy, but the corresponding calculation amount is large, and it needs to consider the surrounding 4*4 points at a time. Based on our compromise, we consider using bilinear interpolation:

[0083] Bilinear interpolation is a quadratic one-dimensional linear interpolation, using four nearest neighbors to estimate the given grayscale value to estimate the pixel value at the new position. It is essentially an extension of one-dimensional linear interpolation. Linear interpolation is a method of estimating the value between two known data points:

[0084] Assume that we know the coordinates (x0, y0) and (x1, y1), and we want to get the value of a position x on the line in the interval [x0, x1]. Figure 2 We can get:

[0085]

[0086] Since x is known, y is:

[0087]

[0088] So, this is a linear interpolation in the x direction.

[0089] Bilinear interpolation is to interpolate in the x-direction and x-direction respectively. It estimates the value of the pixel at the new position based on the values ​​of the four adjacent pixels in the original image. It is an extension of one-dimensional linear interpolation.

[0090] like Figure 3 As shown, assuming that Q is known 11 , Q 12 , Q 21 , Q 22 Four points, to estimate the pixel value f(x,y) at any point (x,y) in the rectangle formed by these four points.

[0091] For two pairs of points Q along the y-axis 11 , Q 21 Linear interpolation in the x direction:

[0092]

[0093] For two pairs of points Q along the y-axis 12 , Q 22 Linear interpolation in the x direction:

[0094]

[0095] Perform linear interpolation in the y direction for two pairs of points R1 and R2 along the x-axis:

[0096]

[0097] At this time, a total of three linear interpolations are performed, and bilinear interpolation only interpolates in the x and y directions instead of performing two interpolations.

[0098] Bilinear interpolation is used to estimate the intensity or color values ​​of pixels in the resized image based on known values ​​in the original image. This method can produce smoother results than nearest neighbor interpolation, which can cause visible artifacts or jagged edges.

[0099] If higher image quality is required, you can consider using other interpolation algorithms, such as cubic interpolation or Lanczos interpolation, but here only bilinear interpolation is needed.

[0100] S200, performing image stitching processing on multispectral images with different viewing angles to obtain first image data;

[0101] It should be noted that, in some embodiments, performing image stitching processing on multispectral images of different perspectives to obtain first image data may include the following steps: using a visual geometry group network to independently encode multispectral images of different perspectives to obtain a feature pyramid corresponding to the multispectral image of each perspective; wherein the feature pyramid is obtained by connecting image features obtained at each level of the visual geometry group network; based on the feature pyramid, performing spatial graph reasoning through a graph convolutional network to obtain the first image data.

[0102] For example, in some specific implementations, image stitching can be implemented as follows: extracting image edge features from the captured / preprocessed images, which are helpful for subsequently stitching and synthesizing images from multiple perspectives. Then, stitching multiple images together into a larger image to create a panoramic image.

[0103] Optionally, in some preferred embodiments, the multispectral image features of different viewpoints are first independently encoded by VGG-16 (Visual Geometry Group Network). Subsequently, the features of the corresponding layers are connected to obtain two feature pyramids with four-scale features, which are used for the reference viewpoint and the target viewpoint respectively. In order to fully extract complementary information from multispectral images, it is necessary to learn powerful representations so as to obtain rich complementary information from multi-level features and achieve more accurate alignment. The essence of multispectral image stitching is to effectively utilize the complementarity of infrared and visible light images to achieve accurate spatial alignment. Therefore, a method based on spatial graph reasoning is adopted, which uses the advantages of graph convolutional networks (GCNs) to simulate feature relationships and effectively extract deformation and integration of multispectral images under different viewpoints. This method embeds multi-scale complementary features of the same view into a set of nodes, and learns the correspondence between different views through powerful dense feature embedding, while developing internal and external correlations to utilize cross-view matching and enhance internal feature differences. By introducing long-range consistency along spatial and channel dimensions, the complementarity of pixel relationships and channel interdependence facilitates the reconstruction of aligned multi-view features, generating informative and reliable wide-field-of-view scenes.

[0104] Since the main goal of the present invention is to create a panoramic image containing information from multiple bands, rather than to enhance specific bands or improve image quality, the priority here is: image stitching first and then image fusion. A large frame is obtained by image stitching, and then the information in the frame is optimized by image fusion.

[0105] Further, the feature pyramid includes a first feature pyramid of a reference viewpoint and a second feature pyramid of a target viewpoint; in some optional implementations, based on the feature pyramid, spatial graph reasoning is performed through a graph convolutional network to obtain the first image data, which may include the following steps: performing multi-scale feature extraction on the first feature pyramid to obtain a first node representation; wherein the multi-scale feature extraction includes a pooling operation, a convolution operation, and an upsampling operation in sequence; the first node representation includes the extracted features corresponding to the features of each scale in the first feature pyramid; performing multi-scale feature extraction on the second feature pyramid to obtain a second node representation; wherein the second node representation includes the extracted features corresponding to the features of each scale in the second feature pyramid; using a progressive mechanism to perform associative reasoning on the extracted features represented by the first node and the extracted features represented by the second node to obtain the guide nodes of each scale corresponding to the reference viewpoint and the target viewpoint; based on the guide nodes, using the connection layer and the fully connected layer to obtain the homography parameters for aligning the reference viewpoint with the target viewpoint; based on the homography parameters, converting the view positions corresponding to the target viewpoint and the reference viewpoint to the same plane, and then obtaining the first image data using the reconstruction and calculation loss function based on the space and channel graph.

[0106] For example, in some specific embodiments, Figure 4 The figure below is a schematic diagram of the hierarchical structure of a graph convolutional network, where max pool is the maximum pooling layer, conv is the convolutional layer, and up sampling is the upsampling layer. Inter and intracorrelation are the mutual and internal correlations of different scales. Concatenation is the connection layer of the guide node, full connection is the full connection layer, and warping is the deformation layer, which converts different view positions into the same plane. Spatial graph reasoning through a graph convolutional network can be implemented through the following process:

[0107] First, the pooling operation is used, followed by the convolution layer (used for convolution operation, corresponding to the subsequent formula Conv) and the upsampling layer (used for upsampling operation), to extract multi-scale features from different viewpoints as the initial node representation, so that a total of four scale features are extracted, and each scale feature is extracted from the image through pooling-convolution-upsampling. Feature n (i) It can be expressed by the following formula:

[0108] n (i) =U(Conv(P(f (i) )) (8)

[0109] Where f() represents the features to be extracted, i is the feature level, P() represents the pooling operation, and U() represents the upper sample layer that ensures that the embedded multi-scale feature maps maintain the same size.

[0110] In order to explicitly infer high-level relationships between infrared and visible light images for better alignment, each node aggregates feature information from all neighboring nodes. Based on the initial embedded nodes obtained above, we use the parameterized edges between nodes to represent spatial correlations and update the initial nodes with this parameter. The messages between any pair of nodes are bidirectional, and the message m lk 、m kl By adjacent node n k 、n l By passing them to each other:

[0111]

[0112] Sig() is the sigmod function, Conv is the convolution layer, and the update of related nodes can be expressed as follows:

[0113]

[0114] Based on the four-scale feature pyramid, a progressive mechanism is used for association reasoning. (In the following parameters, ref and tar represent the reference image and the target image respectively) We start from the minimum scale feature {f ref1 , f tar1}Start (here f ref1 , f tar1 Represents a layer of features of the reference image and the target respectively, 1 here represents the first layer of the four feature scales, that is, the smallest layer. ), the inter and intra correlations of the updated nodes within this scale can be mapped to two view-specific guide nodes n ref g1 , n tar g1 , to merge subsequent node initializations. Specifically, boot node n ref g1 、n tar g1 It can be expressed as:

[0115] n g1 =Conv(Concat(n1,.....n j )),j=N (11)

[0116] Here n g1 Generally refers to n ref g1 、n tar g1 , the two guide nodes are obtained by a unified method. j It is the node after two rounds of internal update and internal update. concat() merges the nodes. Initialize node n (2) {fref2 ,f tar2} is defined as:

[0117] n (2) =Sig(A(n g1 ))*n (2) +n (2) (12)

[0118] n (2) It can be obtained by formula (8), where A() represents global average pooling. In this way, {f ref2 , f tar2 The feature embedding of} incorporates the previous graph information, and the relational reasoning at this scale is the same as that of the previous level.

[0119] By analogy, we can get the guiding nodes n of the 2nd and 3rd layers. ref g2 、n tar g2 、n ref g3 、n tar g3 From these nodes, the homography for multi-spectral multi-view alignment can be derived:

[0120]

[0121] Among them, R() is fully connected, and finally the target feature f is tar4 Change to the reference feature f ref4 , different view positions can be converted to the same plane.

[0122] In multispectral imaging, the present invention fuses the information of infrared and visible light images during the stitching process, ensuring the preservation of the distinct information of the multispectral image. Spatial and channel graph based reconstruction (SCGR) is adopted, which utilizes information from multiple dimensions, including global spatial relationships at the pixel level and global interdependencies at the channel level. In practical applications, the SCGR module consists of spatial graph reasoning (SGR), channel graph reasoning (CGR) and dilated convolution module (DCM).

[0123] Finally, the output image is obtained by calculating the loss (such as L1 loss function).

[0124] S300, performing image fusion processing on the first image data based on a preset image fusion technology to obtain second image data;

[0125] It should be noted that, in some embodiments, performing image fusion processing on the first image data based on a preset image fusion technology to obtain the second image data may include the following steps: performing regional enhancement processing on the first image data based on a regional fusion method to obtain the second image data.

[0126] Exemplarily, in some specific implementations, image fusion can be achieved as follows: using image fusion technology, such as IHS (intensity-chroma-saturation) fusion, PCA (principal component analysis) fusion or deep learning fusion method, and combining the images obtained by perspective stitching to synthesize multi-perspective and multi-spectral images into an image with richer and more comprehensive information.

[0127] Optionally, in some preferred embodiments, since there is already a splicing pattern, a region-based fusion method can be selected. The existing image and the new spectral image are subjected to an image segmentation algorithm (a threshold-based segmentation method is used to save computing resources), and features are extracted and the pixels of the feature area are aligned, and then a direct averaging method is used. Two or more images are simply averaged to generate a fused image.

[0128] S400, using deep learning technology to perform target detection and recognition on the second image data to obtain an abnormality detection result of the target power plant area;

[0129] It should be noted that, in some embodiments, using deep learning technology to perform target detection and recognition on the second image data to obtain anomaly detection results for the target power plant area can include the following steps: using a pre-trained target detection model to perform target detection and recognition on the second image data to obtain anomaly detection results for the target power plant area; wherein the target detection model is trained based on fused image data that has been labeled with anomaly detection labels.

[0130] Exemplarily, in some specific implementations, target detection and recognition can be implemented as follows: deep learning algorithms, such as convolutional neural networks (CNNs), can be used to perform target detection and recognition on the fused images to identify equipment anomalies, hot spots, and other problems in the power plant.

[0131] Optionally, in some preferred implementations, the fused image can be detected after characterization based on the YOLO image target detection scheme to identify various power generation equipment and key locations in the power plant, mainly for fire, overtemperature, and leakage detection of liquids such as oil and water.

[0132] S500: Perform early warning processing in response to the abnormal detection result.

[0133] For example, in some specific implementations, the detection results and warning information are uploaded to the warning system, and an alarm is processed when there is warning information.

[0134] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0135] First of all, it should be noted that the application of multi-view multi-spectral image fusion technology based on drones in power plant early warning detection is the product of the development of drone remote sensing technology, image processing technology, smart power plant construction needs and environmental monitoring technology. The invention background of this technology reflects the trend of multidisciplinary cross-border and technological integration, aiming to provide intelligent and precise solutions for traditional industries.

[0136] Development of UAV remote sensing technology: The application of UAV technology in the field of remote sensing has made significant progress, and UAVs have become an important tool for obtaining surface information. UAV technology includes the establishment of UAV calibration sites, comprehensive verification systems for UAV remote sensing payloads, and UAV remote sensing safety detection technology.

[0137] Demand for smart power plants: With the demand for energy conservation and emission reduction, the thermal power industry faces the challenges of deep peak regulation, ultra-low emissions, and flexible operation. The construction of smart power plants has become an inevitable trend, and it is necessary to introduce new technologies to achieve intelligent upgrades, aiming to improve the intelligence level of power plants through the construction of intelligent infrastructure.

[0138] Innovation in technology integration: Combining UAV remote sensing technology, multispectral image processing technology and intelligent power plant management is an innovative attempt. This technology integration can provide power plants with more efficient and accurate environmental monitoring and equipment early warning services, which helps to improve the safety management level and environmental monitoring capabilities of power plants.

[0139] In view of this, if Figure 5 and Figure 6 As shown, the present invention provides a power plant early warning detection method based on multi-view multi-spectral image fusion on a drone, which can be specifically implemented as follows:

[0140] S1. Data collection: Use drones equipped with multispectral sensors to take aerial photos of the power plant area and obtain image data in different bands. These data can include images in different spectral ranges such as visible light and near infrared, as well as images from different viewing angles to ensure full coverage of the power plant area.

[0141] S2. Image preprocessing: Perform radiation correction and atmospheric correction on the collected multispectral images to eliminate the influence of lighting and atmospheric conditions on image quality. In addition, geometric correction of the images is also required to ensure geometric consistency between images.

[0142] S3, Image stitching: Extract image edge features from the preprocessed images, which are helpful for stitching and synthesizing multi-view images. Then stitch multiple images into a larger image to create a panoramic image.

[0143] S4. Image fusion: Image fusion technology, such as IHS (intensity-chroma-saturation) fusion, PCA (principal component analysis) fusion or deep learning fusion method, is used, and the multi-view and multi-spectral images are combined with the images obtained by perspective stitching to synthesize the multi-view and multi-spectral images into an image with richer and more comprehensive information.

[0144] S5. Target detection and recognition: Use deep learning algorithms, such as convolutional neural networks (CNN), to perform target detection and recognition on the fused images to identify equipment anomalies, hot spots and other problems in the power plant.

[0145] S6. Develop an early warning system: Develop an early warning system that will automatically issue an early warning when a potential problem is detected, reminding power plant managers to take timely measures.

[0146] S7. Continuous monitoring and evaluation: The system can monitor the power plant regularly, collect data and analyze it to evaluate the accuracy of the early warning system and the operating status of the power plant.

[0147] By using drones to collect image data of different bands, the image data can be used to enhance the features of subsequent images. At the same time, images can be collected from multiple angles to solve the problem of incomplete coverage of the monitoring area.

[0148] The specific method to further go to S2 is as follows:

[0149] In the preprocessing stage of the image, the purpose of radiometric correction is to eliminate the non-uniformity of the sensor itself and convert digital counts (DN) into radiometric brightness values. This usually involves using the sensor's radiometric calibration parameters, such as gain and offset, and taking into account factors such as the sun angle and the earth-sun distance. Atmospheric correction aims to eliminate the effects of the atmosphere on the image, including scattering and absorption. This can be done by using radiation transfer models, such as the 6S model, to simulate the effects of the atmosphere on the spectral signal and establish a lookup table of atmospheric correction coefficients. Due to the inherent characteristics of the imaging optical system, as well as the inclination and rotation angles between the lens imaging plane and the object plane, the acquired image has non-linear geometric distortions. These distortions must be eliminated before quantitative analysis and detection of the image.

[0150] In the geometric correction stage, in general, for wide-angle lens imaging, as long as (cubic) polynomial deformation technology and bilinear interpolation are used, this type of distortion can be eliminated and an ideal image can be obtained. Among them, the use of polynomial fitting for spatial coordinate transformation is a key step. In geometric distortion image correction, a reference image is often used to correct the distorted image. Suppose the coordinates of the reference image are (u, v) and the coordinates of the distorted image are (x, y). The two-dimensional polynomial mathematical model used in the resampling method is:

[0151]

[0152] (1) Where n is the number of terms in the polynomial, and aij and bij are the coefficients of each term. Let the reference image be g(u, v) and the distorted image be f(x, y). For the same point in the image, assume that its grayscale value is constant, that is:

[0153] g(u,v)=f(x,y) (2)

[0154] Formula (2) is the basic formula for image geometric distortion correction. X and Y in the reference image are integers, but X and Y calculated by formula (1) are not necessarily integers, and grayscale interpolation operations must be performed.

[0155] Grayscale interpolation methods are commonly used in the nearest neighbor method, bilinear interpolation and cubic convolution. Theoretically, the nearest neighbor method has the worst effect, the cubic convolution method has the best effect, and the bilinear interpolation method is between the two. The cubic convolution method has high calculation accuracy, but the corresponding calculation amount is large, and it needs to consider the surrounding 4*4 points at a time. Based on our compromise, we consider using bilinear interpolation:

[0156] Bilinear interpolation is a quadratic one-dimensional linear interpolation, using four nearest neighbors to estimate the given grayscale value to estimate the pixel value at the new position. It is essentially an extension of one-dimensional linear interpolation. Linear interpolation is a method of estimating the value between two known data points:

[0157] Assume that we know the coordinates (x0, y0) and (x1, y1), and we want to get the value of a position x on the line in the interval [x0, x1]. Figure 2 We can get:

[0158]

[0159] Since x is known, then y is:

[0160]

[0161] So, this is a linear interpolation in the x direction.

[0162] Bilinear interpolation is to interpolate in the x direction and the x direction respectively. It estimates the value of the pixel at the new position based on the values ​​of the four adjacent pixels in the original image. It is an extension of one-dimensional linear interpolation.

[0163] like Figure 3 As shown, assuming that Q is known 11 , Q 12 , Q 21 , Q 22 Four points, to estimate the pixel value f(x,y) at any point (x,y) in the rectangle formed by these four points.

[0164] For two pairs of points Q along the y-axis11 , Q 21 Linear interpolation in the x direction:

[0165]

[0166] For two pairs of points Q along the y-axis 12 , Q 22 Linear interpolation in the x direction:

[0167]

[0168] Perform linear interpolation in the y direction for two pairs of points R1 and R2 along the x-axis:

[0169]

[0170] At this time, a total of three linear interpolations are performed, and bilinear interpolation only interpolates in the x and y directions instead of performing two interpolations.

[0171] Bilinear interpolation is used to estimate the intensity or color values ​​of pixels in the resized image based on known values ​​in the original image. This method can produce smoother results than nearest neighbor interpolation, which can cause visible artifacts or jagged edges.

[0172] If higher image quality is required, you can consider using other interpolation algorithms, such as cubic interpolation or Lanczos interpolation, but here only bilinear interpolation is needed.

[0173] Since the main goal of the present invention is to create a panoramic image containing information from multiple bands, rather than to enhance specific bands or improve image quality, the priority here is: image stitching first and then image fusion. A large frame is obtained by image stitching, and then the information in the frame is optimized by image fusion.

[0174] The specific plan to further go to S3 is as follows:

[0175] The essence of multispectral image stitching is to effectively utilize the complementarity of infrared and visible light images to achieve accurate spatial alignment. Therefore, a method based on spatial graph reasoning is adopted, which takes advantage of graph convolutional networks (GCNs) to model feature relationships and effectively extract deformation and integration of multispectral images under different viewpoints. This method embeds multi-scale complementary features of the same view into a set of nodes, learns the correspondence between different views through powerful dense feature embedding, and develops intra- and inter-correlations to exploit cross-view matching and enhance internal feature differences. By introducing long-range consistency along the spatial and channel dimensions, the complementarity of pixel relationships and channel interdependence facilitates the reconstruction of aligned multi-view features, generating information-rich and reliable wide-field scenes.

[0176] like Figure 4 The figure shows the hierarchical structure of the graph convolutional network, where max pool is the maximum pooling layer, conv is the convolutional layer, and up sampling is the upsampling layer. Inter and intra correlation are the mutual and internal correlations of different scales. Concatenation is the connection layer of the guide node, full connection is the full connection layer, and warping is the deformation layer, which converts different view positions into the same plane. Spatial graph reasoning through the graph convolutional network can be implemented through the following process:

[0177] First, the multispectral image features of different viewpoints are independently encoded through VGG-16 (Visual Geometry Group Network). Subsequently, the features of the corresponding layers are connected to obtain two feature pyramids with four-scale features, which are used for the reference viewpoint and the target viewpoint respectively. In order to fully extract complementary information from multispectral images, it is necessary to learn powerful representations to obtain rich complementary information from multi-level features and achieve more accurate alignment. Using pooling operations, followed by convolution layers (for convolution operations, corresponding to the subsequent formula Conv) and upsampling layers (for upsampling operations), multi-scale features of different viewpoints are extracted as initial node representations, so that a total of four scale features are extracted, and each scale feature is extracted from the image through pooling-convolution-upsampling. Feature n (i) It can be expressed by the following formula:

[0178] n (i) =U(Conv(P(f (i) )) (8)

[0179] Where f() represents the features to be extracted, i is the feature level, P() represents the pooling operation, and U() represents the upper sample layer that ensures that the embedded multi-scale feature maps maintain the same size.

[0180] In order to explicitly infer high-level relationships between infrared and visible light images for better alignment, each node aggregates feature information from all neighboring nodes. Based on the initial embedded nodes obtained above, we use the parameterized edges between nodes to represent spatial correlations and update the initial nodes with this parameter. The messages between any pair of nodes are bidirectional, and the message m lk 、m kl By adjacent node n k 、n l By passing them to each other:

[0181]

[0182] Sig() is the sigmod function, Conv is the convolution layer, and the update of related nodes can be expressed as follows:

[0183]

[0184] Based on the four-scale feature pyramid, a progressive mechanism is used for association reasoning. (In the following parameters, ref and tar represent the reference image and the target image respectively) We start from the minimum scale feature {f ref1 , f tar1}Start (here f ref1 , f tar1 Represents a layer of features of the reference image and the target respectively, 1 here represents the first layer of the four feature scales, that is, the smallest layer. ), the inter and intra correlations of the updated nodes within this scale can be mapped to two view-specific guide nodes n ref g1 , n tar g1 , to merge subsequent node initializations. Specifically, boot node n ref g1 、n tar g1 It can be expressed as:

[0185] n g1 =Conv(Concat(n1,.....n j )),j=N (11)

[0186] Here n g1 Generally refers to n ref g1 、n tar g1 , the two guide nodes are obtained by a unified method. j It is the node after two rounds of internal update and internal update. concat() merges the nodes. Initialize node n (2) {f ref2 ,f tar2} is defined as:

[0187] n (2) =Sig(A(n g1 ))*n (2) +n (2) (12)

[0188] n (2) It can be obtained by formula (8), where A() represents global average pooling. In this way, {f ref2 , f tar2The feature embedding of} incorporates the previous graph information, and the relational reasoning at this scale is the same as that of the previous level.

[0189] By analogy, we can get the guiding nodes n of the 2nd and 3rd layers. ref g2 、n tar g2 、n ref g3 、n tar g3 From these nodes, the homography for multi-spectral multi-view alignment can be derived:

[0190]

[0191] Among them, R() is fully connected, and finally the target feature f is tar4 Change to the reference feature f ref4 , different view positions can be converted to the same plane.

[0192] In multispectral imaging, the present invention fuses the information of infrared and visible light images during the stitching process, ensuring the preservation of the distinct information of the multispectral image. Spatial and channel graph based reconstruction (SCGR) is adopted, which utilizes information from multiple dimensions, including global spatial relationships at the pixel level and global interdependencies at the channel level. In practical applications, the SCGR module consists of spatial graph reasoning (SGR), channel graph reasoning (CGR) and dilated convolution module (DCM).

[0193] Finally, the output image is returned by calculating the loss function.

[0194] The above image stitching technology mainly uses the stitching of infrared light and visible light images, and only two spectral images can be considered. Considering that the images collected by drones may contain more than just infrared light, image fusion enhancement features are performed on the existing results to provide better quality data for target detection.

[0195] The further plan to S4 is as follows:

[0196] Since there is already a mosaic pattern, a region-based fusion method can be selected. The existing image and the new spectral image are subjected to an image segmentation algorithm (a threshold-based segmentation method is used to save computing resources), and features are extracted and the pixels in the feature area are aligned before using a direct averaging method. Two or more images are simply averaged to generate a fused image.

[0197] The further plan to S5 is as follows:

[0198] You can choose to detect the fused image after characterization processing based on the YOLO image target detection solution, and identify the various power generation equipment and key locations of the power plant. It is mainly aimed at detecting fire, overtemperature, and leakage of liquids such as oil and water.

[0199] The detection results and warning information are uploaded to the early warning system, and alarm processing is performed when there is warning information.

[0200] Among them, in some optional implementation scenarios, the technical solution of the present invention can also apply the following alternative solutions:

[0201] (i) Satellite remote sensing technology: Using satellite-borne remote sensing sensors to monitor power plants can cover a wider area, but may not be as good as drones in terms of resolution and real-time performance.

[0202] (ii) Ground sensor network: Ground sensors, such as temperature, humidity, and gas sensors, are deployed around the power plant to monitor environmental changes in real time, but they cannot provide the high viewing angle and wide coverage that drones offer.

[0203] (iii) Fixed camera monitoring system: Cameras are installed at key locations in the power plant for 24-hour monitoring, but are also limited by viewing angle and monitoring range, and may require more manual intervention.

[0204] (iv) Multi-sensor fusion technology: In addition to multispectral images, other types of sensor data can also be fused, such as infrared thermal images, laser radar (LiDAR), etc. to provide richer monitoring information.

[0205] (V) Cloud computing and the Internet of Things (IoT): Cloud computing platforms are used to process and analyze large amounts of data from drones and other sensors. IoT technology can collect and transmit data in real time, improving the efficiency and accuracy of monitoring.

[0206] These alternatives can be selected and adjusted according to the specific needs, budget and technical preferences of the power plant to achieve the best monitoring and early warning effects.

[0207] In summary, the power plant early warning detection method based on multi-view multi-spectral image fusion on a drone provided by the present invention can obtain conventional images and multi-spectral images through the multi-spectral camera on the drone, and because the drone can move at will, we can get images from multiple different perspectives, and fuse these images through image processing methods such as OpenCV and deep learning algorithms. Expand the field of view of the scene and improve the overall picture and information content of the image. It can also carry multi-source sensors to collect more information, combine multiple information categories and apply this technology to early warning detection of power plant equipment.

[0208] The technical solution of the present invention can be widely used in the fields of real-time monitoring, landscape photography, etc., providing users with a broader and more realistic field of view and image experience, as well as a more comprehensive and intelligent monitoring method.

[0209] The power plant early warning detection method based on multi-view and multi-spectral image fusion on drones is a multidisciplinary field that integrates research results and application practices in multiple technical fields such as remote sensing technology, image processing, machine learning, data fusion, early warning system development, automatic control, and information technology.

[0210] Remote sensing technology: This method uses multispectral sensors carried by drones for remote sensing monitoring to obtain image data of the power plant and its surrounding environment. This data can be used to analyze the thermal imaging, vegetation conditions, land use, etc. of the power plant, thereby evaluating the operating environment of the power plant.

[0211] Image processing and analysis: Through image fusion technology, images of different perspectives and different spectral ranges are combined to obtain clearer and richer image information. This usually involves image preprocessing, feature extraction, image registration and fusion algorithms.

[0212] Machine Learning and Artificial Intelligence: Based on image fusion, machine learning algorithms are applied to analyze the fused images to automatically identify and classify abnormal situations in the power plant, such as equipment failure, hot spot detection, etc.

[0213] Data fusion technology: Multi-source sensor data fusion is a key technology for processing and analyzing multiple sensor data carried by drones. It realizes the fusion and utilization of multi-sensor data through the process of detection, association, combination, and estimation, and obtains accurate drone status and target information to provide support for decision-making.

[0214] Early warning system development: Based on drone monitoring data and machine learning analysis results, develop a power plant early warning system to achieve early identification and warning of potential risks.

[0215] Automation and Intelligent Control: This approach may also involve automation and intelligent control technologies to optimize the operation and maintenance processes of the power plant.

[0216] Environmental Monitoring and Protection: This method contributes to environmental protection and sustainable development by monitoring the impact of power plants on the surrounding environment.

[0217] Information Technology: Data collection, processing, storage and transmission all rely on modern information technology, including database management, cloud computing and big data analysis.

[0218] Specifically, the power plant early warning detection method based on multi-view multi-spectral image fusion on drones provided by the present invention is mainly used for regular inspection and fault diagnosis of power plant equipment, monitoring and evaluation of environmental impact of power plants, power plant safety management and risk control, optimization of power plant operating efficiency and cost control, etc. The implementation of this method helps to improve the intelligence level of power plants and achieve more efficient, safer and more environmentally friendly electricity production. It aims to solve the following technical problems:

[0219] (i) Improve monitoring efficiency and accuracy: Traditional power plant monitoring methods may be inefficient and difficult to cover all areas. The multispectral sensor carried by drones can quickly obtain images of large areas of power plant areas, and improve monitoring efficiency and accuracy through image fusion technology.

[0220] (ii) Real-time monitoring and fault warning: By analyzing the data collected by drones in real time, abnormal conditions of power plant equipment, such as abnormal temperature and equipment wear, can be discovered in a timely manner, thus achieving early warning of faults.

[0221] (iii) Environmental impact assessment: Multispectral images can be used to assess the impact of power plants on the surrounding environment, such as emission monitoring and environmental change analysis, to help power plants take appropriate environmental protection measures.

[0222] (iv) Data integration and analysis: Integrate the data collected by drones with the power plant’s existing information system to provide deeper insights and decision support through big data analysis and machine learning technology.

[0223] (V) Intelligence and Automation: This method reduces human intervention and improves the objectivity and accuracy of monitoring through automated image acquisition and analysis processes.

[0224] like Figure 7 As shown, the embodiment of the present invention further provides a power plant early warning detection device 900, which may include:

[0225] The first module 901 is used to obtain image data of the target power plant area; the image data includes multi-spectral images at different viewing angles;.

[0226] The second module 902 is used to perform image stitching processing on multispectral images with different viewing angles to obtain first image data;

[0227] The third module 903 is used to perform image fusion processing on the first image data based on a preset image fusion technology to obtain second image data;

[0228] The fourth module 904 is used to perform target detection and recognition on the second image data using deep learning technology to obtain an abnormality detection result of the target power plant area;

[0229] The fifth module 905 is used to perform early warning processing in response to the abnormal detection result.

[0230] In some embodiments, the apparatus may further include:

[0231] The sixth module is used for preprocessing the image data;

[0232] The image stitching process is performed based on the pre-processed image data.

[0233] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0234] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned power plant early warning detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0235] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0236] See also Figure 8 , Figure 8 The hardware structure of an electronic device 1000 of another embodiment is illustrated. The electronic device 1000 includes:

[0237] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0238] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 is called to execute the power plant early warning detection method of the embodiment of the present invention;

[0239] Input / output interface 1003, used to implement information input and output;

[0240] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0241] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0242] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0243] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned power plant early warning detection method is implemented.

[0244] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0245] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0246] The power plant early warning detection method, power plant early warning detection device, electronic device and storage medium provided by the embodiment of the present invention obtain image data of the target power plant area; the image data includes multispectral images at different viewing angles; the multispectral images at different viewing angles are subjected to image stitching processing to obtain first image data; the first image data is subjected to image fusion processing based on a preset image fusion technology to obtain second image data; the second image data is subjected to target detection and recognition using deep learning technology to obtain abnormal detection results of the target power plant area; and early warning reminder processing is performed in response to the abnormal detection results. The embodiment of the present invention can be applied to regular inspection and fault diagnosis of power plant equipment, monitoring and evaluation of environmental impact of power plants, safety management and risk control of power plants, optimization of power plant operating efficiency and cost control, etc. based on the acquired image data of power plants. The present invention can efficiently and conveniently realize early warning detection of power plants.

[0247] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0248] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0249] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0250] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0251] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0252] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0253] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0254] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0256] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0257] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A power plant early warning detection method, characterized in that: The method comprises the following steps: Acquire image data of a target power plant area; the image data includes multispectral images at different viewing angles; Performing image stitching processing on the multispectral images at different viewing angles to obtain first image data; Performing image fusion processing on the first image data based on a preset image fusion technology to obtain second image data; Using deep learning technology to perform target detection and recognition on the second image data to obtain an abnormality detection result of the target power plant area; An early warning reminder process is performed in response to the abnormal detection result.

2. The power plant early warning detection method according to claim 1, characterized in that: Before the step of performing image stitching processing on the multispectral images at different viewing angles, the method further comprises the following steps: Preprocessing the image data; Wherein, the image stitching process is performed based on the pre-processed image data.

3. The power plant early warning detection method according to claim 2, characterized in that: The image data is collected based on a preset multi-spectral sensor; the image data is preprocessed, Includes at least one of the following steps: Performing radiometric correction on the image data based on radiometric calibration parameters of the multispectral sensor; Performing atmospheric correction on the image data based on a preset radiation transfer model simulation; The image data is geometrically corrected using polynomial deformation techniques and bilinear interpolation methods.

4. The power plant early warning detection method according to claim 1, characterized in that: The step of performing image stitching processing on the multispectral images at different viewing angles to obtain first image data comprises the following steps: Using a visual geometry group network to independently encode the multispectral images at different viewing angles, and obtain a feature pyramid corresponding to the multispectral image at each viewing angle; Wherein, the feature pyramid is obtained by connecting the image features obtained at each level of the visual geometry group network; Based on the feature pyramid, spatial graph reasoning is performed through a graph convolutional network to obtain the first image data.

5. The power plant early warning detection method according to claim 4, characterized in that: The feature pyramid includes a first feature pyramid of a reference viewpoint and a second feature pyramid of a target viewpoint; performing spatial graph reasoning through a graph convolutional network based on the feature pyramid to obtain the first image data includes the following steps: Performing multi-scale feature extraction on the first feature pyramid to obtain a first node representation; The multi-scale feature extraction includes a pooling operation, a convolution operation, and an upsampling operation in sequence; the first node represents an extracted feature corresponding to a feature of each scale in the first feature pyramid; Performing the multi-scale feature extraction on the second feature pyramid to obtain a second node representation; The second node represents an extracted feature corresponding to a feature at each scale in the second feature pyramid; Using a progressive mechanism to perform associative reasoning on the extracted features represented by the first node and the extracted features represented by the second node, to obtain a guide node of each scale corresponding to the reference viewpoint and the target viewpoint; Based on the guide node, a connection layer and a fully connected layer are used to process and obtain homography parameters for aligning the reference viewpoint with the target viewpoint; Based on the homography parameters, the view positions corresponding to the target viewpoint and the reference viewpoint are converted into the same plane, and then the first image data is obtained by using a reconstruction based on a space and channel map and a calculation loss function.

6. The power plant early warning detection method according to claim 1, characterized in that: The step of performing image fusion processing on the first image data based on a preset image fusion technology to obtain second image data comprises the following steps: The first image data is subjected to regional enhancement processing based on a region fusion method to obtain the second image data.

7. The power plant early warning detection method according to claim 1, characterized in that: The method of performing target detection and recognition on the second image data by using deep learning technology to obtain an abnormality detection result of the target power plant area includes the following steps: Using a pre-trained target detection model to perform target detection and recognition on the second image data to obtain an abnormality detection result of the target power plant area; The target detection model is trained based on fused image data that has been labeled with anomaly detection labels.

8. A power plant early warning detection device, characterized in that: The device comprises: The first module is used to obtain image data of the target power plant area; the image data includes multispectral images at different viewing angles;. The second module is used to perform image stitching processing on the multispectral images of different viewing angles to obtain first image data; A third module is used to perform image fusion processing on the first image data based on a preset image fusion technology to obtain second image data; A fourth module is used to perform target detection and recognition on the second image data using deep learning technology to obtain an abnormality detection result of the target power plant area; The fifth module is used to perform early warning and reminder processing in response to the abnormal detection result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images