Image processing method and device, computer device and storage medium

Abstract

The application relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring multiple images of a picture under different exposure times; each image corresponds to an exposure time; determining a target region in which a motion target exists in each image based on a motion target detection model; determining a weight value corresponding to each pixel point in each image according to each image and the corresponding target region; the weight value is used for representing the correlation degree between the pixel point and the motion target; correcting each image based on the weight value corresponding to each pixel point in each image to obtain a corrected image corresponding to each image; and fusing the corrected images corresponding to the images to obtain a target image corresponding to the picture. The method can improve the reliability of fault detection and diagnosis of a power transmission line.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In the power industry, especially during the inspection of power transmission lines, it is common to rely on equipment such as drones or surveillance cameras to collect images, and then use these images to detect and diagnose faults in the transmission lines.

[0003] In related technologies, in order to enable images to reflect as much information as possible, the Multi Exposure Fusion (MEF) algorithm is often used to fuse multiple images of the same scene with different exposure times to obtain a fused image with high information content. Then, the fused image is used for fault detection and diagnosis of power transmission lines.

[0004] However, multi-scale exposure image fusion algorithms require that there be no moving targets in the image. Once a moving target is present, ghosting will occur to some extent during the fusion process, which will interfere with the detection and diagnosis of actual faults and make the reliability of fault detection and diagnosis of power transmission lines low. Summary of the Invention

[0005] Therefore, it is necessary to provide an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the reliability of fault detection and diagnosis of transmission lines, addressing the technical problem of low reliability in the aforementioned fault detection and diagnosis of transmission lines.

[0006] In a first aspect, this application provides an image processing method, comprising:

[0007] Acquire multiple images of the scene at different exposure times; each image corresponds to one exposure time;

[0008] Based on the moving target detection model, the target region containing moving targets in each image is determined;

[0009] Based on each image and its corresponding target region, a weight is determined for each pixel in each image; the weight is used to characterize the degree of association between the pixel and the moving target.

[0010] Based on the weight corresponding to each pixel in each image, each image is corrected to obtain a corrected image corresponding to each image;

[0011] By fusing the corrected images corresponding to each image, the target image corresponding to the scene is obtained.

[0012] In one of the embodiments, the determining of the weight value corresponding to each pixel point in each image based on the image and the corresponding target region comprises:

[0013] For each image, the pixel value and the brightness of each pixel point in the image are determined, and the positional relationship between the pixel point and the target region in the image is determined;

[0014] Based on the pixel value and the brightness of each pixel point, and the positional relationship between the pixel point and the target region in the image, the weight value corresponding to the pixel point is determined.

[0015] In one of the embodiments, the correcting of each image based on the weight value corresponding to each pixel point in the image to obtain the corrected image corresponding to the image comprises:

[0016] For each image, the pixel value of each pixel point in the image is corrected based on the weight value corresponding to the pixel point to obtain the corrected pixel value corresponding to the pixel point;

[0017] Based on the corrected pixel value corresponding to each pixel point, the corrected image corresponding to the image is obtained.

[0018] In one of the embodiments, the resolutions of the multiple images are the same, and the resolutions of the corrected images corresponding to the multiple images are the same;

[0019] The fusing of the corrected images corresponding to the multiple images to obtain the target image corresponding to the picture comprises:

[0020] The corrected pixel values corresponding to the same pixel point in the multiple corrected images are fused to obtain the target pixel value corresponding to the pixel point; the same pixel point is a pixel point with the same position in the multiple corrected images;

[0021] Based on the target pixel value corresponding to each pixel point, the target image corresponding to the picture is obtained.

[0022] In one of the embodiments, the determining of the target region in each image where a moving target exists based on the moving target detection model comprises:

[0023] For each image, the image is input into the moving target detection model to determine multiple candidate regions associated with a moving target in the image, and the confidence of the multiple candidate regions; the confidence is used to represent the possibility of the candidate region existing the moving target;

[0024] screening a target region in which the moving target exists from the plurality of candidate regions based on a confidence of each candidate region, and determining a position of the target region in the image.

[0025] In one of the embodiments, the picture is a picture centered on the power transmission line.

[0026] After fusing the corrected images corresponding to the images to obtain the target image corresponding to the picture, the method further comprises:

[0027] inputting the target image into a fault detection and diagnosis model corresponding to the power transmission line to obtain a fault detection and diagnosis result of the power transmission line.

[0028] In a second aspect, the present application further provides an image processing device, comprising:

[0029] an image acquisition module configured to acquire a plurality of images of a picture under different exposure times, each image corresponding to an exposure time;

[0030] a target detection module configured to determine a target region in which a moving target exists in each image based on a moving target detection model;

[0031] a weight determination module configured to determine a weight corresponding to each pixel point in each image according to the each image and the corresponding target region, the weight being used to represent a correlation degree between the pixel point and the moving target;

[0032] an image correction module configured to correct each image based on the weight corresponding to each pixel point in the each image to obtain a corrected image corresponding to the each image;

[0033] an image fusion module configured to fuse the corrected images corresponding to the images to obtain a target image corresponding to the picture.

[0034] In a third aspect, the present application further provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0035] acquiring a plurality of images of a picture under different exposure times, each image corresponding to an exposure time;

[0036] determining a target region in which a moving target exists in each image based on a moving target detection model;

[0037] determining a weight corresponding to each pixel point in each image according to the each image and the corresponding target region, the weight being used to represent a correlation degree between the pixel point and the moving target;

[0038] correct the each image based on the weight corresponding to each pixel point in the each image, to obtain a corrected image corresponding to the each image;

[0039] fuse the corrected images corresponding to the images to obtain a target image corresponding to the picture.

[0040] In a fourth aspect, the present application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0041] obtain multiple images of a picture under different exposure times, each image corresponding to an exposure time;

[0042] determine a target region in which a moving target exists in each image based on a moving target detection model;

[0043] determine a weight corresponding to each pixel point in the each image according to the each image and the corresponding target region, the weight being used to represent a correlation degree between the pixel point and the moving target;

[0044] correct the each image based on the weight corresponding to each pixel point in the each image, to obtain a corrected image corresponding to the each image;

[0045] fuse the corrected images corresponding to the images to obtain a target image corresponding to the picture.

[0046] In a fifth aspect, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0047] obtain multiple images of a picture under different exposure times, each image corresponding to an exposure time;

[0048] determine a target region in which a moving target exists in each image based on a moving target detection model;

[0049] determine a weight corresponding to each pixel point in the each image according to the each image and the corresponding target region, the weight being used to represent a correlation degree between the pixel point and the moving target;

[0050] correct the each image based on the weight corresponding to each pixel point in the each image, to obtain a corrected image corresponding to the each image;

[0051] fuse the corrected images corresponding to the images to obtain a target image corresponding to the picture.

[0052] The image processing method, device, computer device, storage medium and computer program product obtain multiple images of a picture under different exposure times, each image corresponding to an exposure time; determine a target region in which a moving target exists in each image based on a moving target detection model; determine a weight value corresponding to each pixel point in each image according to each image and the corresponding target region; the weight value is used to represent the correlation degree between the pixel point and the moving target; correct each image based on the weight value corresponding to each pixel point in each image to obtain a corrected image corresponding to each image; and finally fuse the corrected images corresponding to the images to obtain a target image corresponding to the picture. In this way, based on the multiple images of the picture under different exposure times, the details of different brightness levels of the picture can be captured, thereby helping to improve the dynamic range of the image and making the details clearer and more visible; based on the moving target detection model, the target region in which the moving target exists in each image can be determined, and then each image can be corrected with respect to the moving target to eliminate ghosting generated in the fusion process to obtain the corrected image corresponding to each image; based on the fusion processing of the corrected images, the target image with high information content and without ghosting can be obtained, thereby improving the reliability of fault detection and diagnosis of the power transmission line. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0054] Figure 1 A flowchart of an image processing method in an embodiment;

[0055] Figure 2 A flowchart of a step of determining a weight value corresponding to each pixel point in each image according to each image and the corresponding target region in an embodiment;

[0056] Figure 3 A flowchart of a step of correcting each image based on the weight value corresponding to each pixel point in each image to obtain a corrected image corresponding to each image in an embodiment;

[0057] Figure 4 A flowchart of a step of fusing the corrected images corresponding to the images to obtain a target image corresponding to the picture in an embodiment;

[0058] Figure 5 A flowchart of an image processing method in another embodiment;

[0059] Figure 6 Structure block diagram of an image processing device in an embodiment;

[0060] Figure 7 Internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] In an exemplary embodiment, as shown in Figure 1 An image processing method is provided, and the present embodiment is exemplified by the method applied to a server. It should be understood that the method can also be applied to a terminal, and can also be applied to a system including a server and a terminal, and is realized through the interaction of the server and the terminal. The server can be realized by an independent server or a server cluster composed of multiple servers, and the terminal can be, but is not limited to, various personal computers, notebook computers, smart phones and tablet computers, etc. In the present embodiment, the method includes the following steps:

[0064] Step S102, acquiring multiple images of a picture under different exposure times.

[0065] Each image corresponds to an exposure time; the images under different exposure times reflect the details of different brightness levels of the picture.

[0066] Further, the number of multiple images is three.

[0067] Specifically, the server acquires multiple images of the same picture under different exposure times through a shooting device, such as a camera.

[0068] Step S104, determining a target region in which a moving target exists in each image based on a moving target detection model.

[0069] The moving target detection model is used to detect moving targets in images; the moving target detection model can be realized by a target detection model.

[0070] Wherein, the moving target refers to an object in a moving state; it can be understood that in the related art, the ghost in the fusion process is caused by the moving target, and therefore the moving area is an area in which the ghost phenomenon is likely to occur in the subsequent image fusion process, and therefore, the moving target is detected first in the present application, and the moving target is processed, so as to eliminate the ghost caused by the moving target in the fusion process.

[0071] Further, the moving target detection model is constructed by using a Faster R-CNN algorithm (an algorithm combining RPN and Fast R-CNN, RPN, Region Proposal Network, Fast R-CNN, Fast Regions with ConvNet, Fast Region Convolutional Neural Network). The moving target detection model includes an RPN module and a Fast R-CNN module. The RPN module is used to generate a region of interest, and the Fast R-CNN module includes a classifier and a regressor. The classifier is used to determine the target type, and the regressor is used to perform boundary regression. In the image processing method provided by the present application, the region of interest refers to a region associated with the moving target, such as a region suspected to contain a moving target; and the target type refers to whether the region contains a moving target.

[0072] Specifically, the server inputs each image into the moving target detection model, detects whether there is a moving target in each image through the moving target detection model, and determines the target region containing the moving target and the position of the target region in the image.

[0073] For example, the moving target detection model first extracts features from the image, then determines whether there is a moving target in the image based on the features of the image domain, and uses a target frame to frame the target region containing the moving target in the image; then, the moving target detection model constructs a corresponding two-dimensional coordinate system based on the image, determines the position of the target region in the image based on the coordinates of the target frame in the two-dimensional coordinate system; and then the moving target detection model outputs the image including the target frame and the coordinates; the server can determine the target region containing the moving target in the image and the position of the target region in the image through the output of the moving target detection model.

[0074] In step S106, the weight value corresponding to each pixel point in each image is determined according to each image and the corresponding target region.

[0075] Wherein, the weight value is used to represent the degree of association between the pixel point and the moving target.

[0076] Specifically, for each image, the server determines the degree of association between each pixel in the image and the moving target based on the image and the target region in the image, and uses weights to represent the degree of association, thereby obtaining the weight corresponding to each pixel in the image.

[0077] Furthermore, the server maps the image to a corresponding two-dimensional coordinate system, meaning that each pixel in the image has corresponding pixel coordinates in the two-dimensional coordinate system; then, the server maps the weights corresponding to each pixel in the image to a two-dimensional coordinate system to obtain the image's weight map, meaning that each pixel has corresponding weight coordinates in its weight map.

[0078] For example, image number Line number The pixel coordinates of the column in the two-dimensional coordinate system are: The weight corresponding to this pixel in the weight map is also its weight coordinates. .

[0079] Step S108: Based on the weight corresponding to each pixel in each image, each image is corrected to obtain the corrected image corresponding to each image.

[0080] Specifically, for each image, the server corrects the pixel value of each pixel in the image based on the weight corresponding to each pixel in the image, so as to obtain the corrected image corresponding to each image.

[0081] For example, let's record the first... In the image, the pixel coordinates are The pixel value of the pixel is The weight corresponding to this pixel is So the server uses Correction The corrected pixel value corresponding to this pixel is obtained. This leads to the corrected image corresponding to the given image.

[0082] Step S110: Fuse the corrected images corresponding to each image to obtain the target image corresponding to the scene.

[0083] Specifically, the server performs weighted fusion of the corrected images corresponding to each image to obtain the target image corresponding to the scene.

[0084] For example, let's record the first... The corrected image corresponding to the image is Then the target image We obtain the following from Formula 1:

[0085] (Formula 1)

[0086] wherein, denotes the total number of images; denotes the corresponding to the fusion weight of the corresponding modified image of each image; the fusion weight corresponding to each modified image can be flexibly set according to actual needs; for example, the longer the corresponding exposure time of the modified image, the greater the corresponding fusion weight; for another example, the fusion weights corresponding to the modified images under different exposure times are the same.

[0087] In the image processing method, the server first acquires a plurality of images of a picture under different exposure times; each image corresponds to an exposure time; then determines a target region in which a motion target exists in each image based on a motion target detection model; then determines a weight corresponding to each pixel point in each image according to each image and the corresponding target region; the weight is used to represent the correlation degree between the pixel point and the motion target; then modifies each image based on the weight corresponding to each pixel point in each image to obtain a modified image corresponding to each image; finally, fuses the modified images corresponding to each image to obtain a target image corresponding to the picture. In this way, based on the plurality of images of the picture under different exposure times, the server can capture the details of different brightness levels of the picture, thereby helping to improve the dynamic range of the image and making the details clearer and more visible; based on the motion target detection model, the server can determine the target region in which the motion target exists in each image, and then can modify each image with respect to the motion target to eliminate the ghosting generated in the fusion process to obtain the modified image corresponding to each image; based on the fusion processing of each modified image, the server can obtain a target image with high information content and without ghosting, thereby improving the reliability of fault detection and diagnosis of the power transmission line.

[0088] As Figure 2 shown, in an exemplary embodiment, the step S106 of determining the weight corresponding to each pixel point in each image according to each image and the corresponding target region specifically includes the following steps:

[0089] Step S202, for each image, determining the pixel value and brightness of each pixel point in the image, and determining the positional relationship between each pixel point and the target region in the image.

[0090] Step S204, based on the pixel value and brightness of each pixel point, and the positional relationship between each pixel and the target region in the image, determining the weight corresponding to each pixel point.

[0091] The positional relationship between each pixel and the target region in the image includes whether the pixel falls within the target region; if the pixel does not fall within the target region, the positional relationship also includes the distance between the pixel and the target region. Furthermore, if there are multiple target regions in the image, the distance between the pixel and each target region is the shortest distance among all the distances between the pixel and each target region.

[0092] Specifically, for each image, the server first determines the pixel value and brightness of each pixel in the image, and simultaneously determines whether each pixel falls within the target region of the image. If it does not fall within the target region, the server further determines the distance between the pixel and the target region. Since moving targets usually cause large changes in pixel values ​​between adjacent pixels, and the brightness of moving targets also varies, the server determines the rate of change of pixel values ​​between each pixel based on the pixel value of each pixel, such as the gradient or rate of change of pixel values. Then, based on the rate of change of pixel values ​​between each pixel, the brightness of each pixel, and the positional relationship between each pixel and the target region, the server determines the weight corresponding to each pixel.

[0093] Among them, the more relevant the positional relationship between the pixel and the target area (the closer it is to the target area), the faster the pixel value changes, and the greater the brightness, the greater the corresponding weight.

[0094] In this embodiment, the server comprehensively evaluates the correlation between each pixel and the moving target based on the pixel value, brightness, and positional relationship between the pixel and the target area, and thus assigns a corresponding weight to each pixel.

[0095] like Figure 3 As shown, in an exemplary embodiment, step S108, which corrects each image based on the weight corresponding to each pixel in each image to obtain a corrected image for each image, specifically includes the following steps:

[0096] Step S302: For each image, based on the weight corresponding to each pixel in the image, the pixel value of each pixel is corrected to obtain the corrected pixel value corresponding to each pixel.

[0097] Step S304: Based on the corrected pixel value corresponding to each pixel, obtain the corrected image corresponding to the image.

[0098] Specifically, for each image, the server takes the weight value corresponding to each pixel point in the image as the pixel value correction coefficient corresponding to each pixel point; then, for each pixel point, the server corrects the pixel value of the pixel point according to the pixel value correction coefficient corresponding to the pixel point, to obtain the corrected pixel value corresponding to the pixel point; and then, the server obtains the corrected image corresponding to the image based on the corrected pixel value corresponding to each pixel point.

[0099] For example, for each pixel point, the server takes the product of the pixel value of the pixel point and the corresponding pixel value correction coefficient as the corrected pixel value corresponding to the pixel point, as shown in formula 2:

[0100] (Formula 2)

[0101] wherein, represents the corrected pixel value corresponding to the pixel point with the pixel point coordinate of (i, j) in the i-th image; represents the pixel value of the pixel point with the pixel point coordinate of (i, j) in the i-th image; represents the weight value corresponding to the pixel point with the pixel point coordinate of (i, j) in the i-th image.

[0102] In this embodiment, the server can eliminate the ghost generated in the fusion process of the moving target in advance by correcting the pixel value of each pixel point through the weight value corresponding to the pixel point, reduce the influence of the moving target on the image quality, and further obtain a target image with high information content and no ghost, thereby improving the reliability of fault detection and diagnosis of the power transmission line.

[0103] In an exemplary embodiment, the resolutions of the multiple images are the same, and the resolutions of the corrected images corresponding to the multiple images are the same.

[0104] Therefore, the number and arrangement of the pixel points contained in the multiple images are the same, and the number and arrangement of the pixel points contained in the corrected images corresponding to the multiple images are also the same.

[0105] As shown in FIG. 4, the above step S110 includes the following steps: Figure 4

[0106] Step S402: performing fusion processing on the corrected pixel values corresponding to the same pixel point in each corrected image to obtain the target pixel value corresponding to the pixel point.

[0107] ​​​​​​​Step S404: obtaining the target image corresponding to the picture based on the target pixel value corresponding to each pixel point.

[0108] The same pixel point is a pixel point with the same position in each of the correction images, i.e., a pixel point with the same coordinate in the two-dimensional coordinate system corresponding to each of the correction images.

[0109] Specifically, the server takes the fusion weight corresponding to each correction image as the fusion coefficient corresponding to the correction image, and then, for each pixel point with the same position in each of the correction images, performs fusion processing on the correction pixel values corresponding to the pixel point in each of the correction images to obtain the target pixel value corresponding to the same pixel point, thereby obtaining the target image corresponding to the picture.

[0110] For example, for each pixel point, the server takes the product of each correction pixel value corresponding to the pixel point and the corresponding fusion coefficient as the target pixel value corresponding to the pixel point, as shown in Formula 3:

[0111] (Formula 3)

[0112] Further, the fusion weight corresponding to each correction image is the same and is 1, and therefore, the calculation formula of the target pixel value can be represented by Formula 4:

[0113] (Formula 4)

[0114] In this embodiment, the server fuses the correction pixel values corresponding to the same pixel point in each of the correction images through the fusion weight corresponding to each correction image, which can realize the fusion processing of multiple correction images in the case of eliminating the ghost generated by the moving target in the fusion process in advance, and further can obtain a target image with high information content and without ghost, thereby improving the reliability of fault detection and diagnosis of the power transmission line.

[0115] In an example embodiment, the step S104 of determining the target region with the moving target in each image based on the moving target detection model specifically includes the following contents: for each image, inputting the image into the moving target detection model to determine a plurality of candidate regions associated with the moving target in the image and the confidence of the plurality of candidate regions; and based on the confidence of each candidate region, screening the target region with the moving target from the plurality of candidate regions and determining the position of the target region in the image.

[0116] The confidence is used to represent the possibility of the candidate region having the moving target.

[0117] Specifically, the server first inputs each image into a convolution residual network ResNet101 (Residual Network-101, a deep convolutional neural network architecture) to realize feature extraction of the image; then, the server inputs the image and the image features into a motion target detection model constructed using a Faster R-CNN algorithm, an RPN module in the motion target detection model first determines whether the image contains a motion target, and uses a candidate box to frame a candidate region associated with the motion target and outputs the confidence of each candidate region, then the RPN module uses non-maximum suppression to filter multiple candidate regions, retains at least one candidate region that meets the confidence requirement and does not overlap with each other, and inputs multiple candidate regions that meet the confidence requirement and do not overlap with each other into an R-CNN module in the motion target detection model; a classifier in the R-CNN module further screens out a target region in which the motion target exists from the multiple candidate regions that meet the confidence requirement and do not overlap with each other according to the confidence, and a regressor in the R-CNN module performs boundary fine-tuning on a target frame corresponding to the target region to more accurately fit the real position of the motion target.

[0118] Further, in the training process of the motion target detection model, the present application has the following improvements:

[0119] First, the server acquires sample images containing motion targets under different exposure times, and uses a K-means (k-means clustering algorithm) clustering method to preprocess the length, width and aspect ratio of anchor boxes of the sample images.

[0120] Second, the ROI Pooling (Region of Interest Pooling) layer in the traditional Faster R-CNN network will crop the preselected frame of images of different sizes into a fixed size feature map, and take the integer twice in the entire network framework, resulting in a deviation of the result frame after down-sampling from the original image, and the impact on smaller motion targets in the image is more prominent. Therefore, the present application uses ROI Align (Region of Interest Align) to keep the floating-point boundaries of each candidate region, thereby solving the above deviation problem. Specifically, the bilinear interpolation method is used to fix the candidate region to a feature map size of 14x14.

[0121] In this embodiment, the server can perform target detection processing corresponding to the moving target on the image through the moving target detection model, so as to determine the target region in which the moving target exists in the image, and facilitate subsequent processing of the moving target to eliminate ghosting generated in the fusion process.

[0122] In an example embodiment, the picture is a picture centered on the power transmission line.

[0123] After obtaining the target image corresponding to the picture by fusing the corrected images corresponding to the images in step S110, the following is further included: inputting the target image into a fault detection and diagnosis model corresponding to the power transmission line to obtain a fault detection and diagnosis result of the power transmission line.

[0124] Specifically, based on the target image, the server can perform corresponding fault detection and diagnosis. Taking fault detection and diagnosis of the power transmission line as an example, the picture is a picture centered on the power transmission line, the multiple images are images of the power transmission line and its surroundings under different exposure times, and the target image is a target image of the power transmission line and its surroundings. After obtaining the target image of the power transmission line and its surroundings, the server inputs the target image into a fault detection and diagnosis model corresponding to the power transmission line to perform fault detection and diagnosis on the operation of the power transmission line, thereby obtaining a fault detection and diagnosis result of the power transmission line.

[0125] In this embodiment, based on the target image of the power transmission line with high information content, the server can realize fault detection and diagnosis of the power transmission line, and based on the target image of the power transmission line without ghosting, the server can improve the reliability of the fault detection and diagnosis of the power transmission line.

[0126] In an example embodiment, as shown in Figure 5 Another image processing method is provided, which is applied to a server as an example for illustration, and includes the following steps:

[0127] Step S501: Obtain multiple images of a picture under different exposure times.

[0128] Step S502: For each image, input the image into a moving target detection model to determine multiple candidate regions associated with the moving target in the image and confidence of the multiple candidate regions.

[0129] Step S503: Based on the confidence of each candidate region, filter out a target region in which the moving target exists from the multiple candidate regions, and determine a position of the target region in the image.

[0130] Step S504: For each image, determine a pixel value and brightness of each pixel point in the image, and determine a positional relationship between each pixel point and the target region in the image.

[0131] In step S505, the weight value corresponding to each pixel point is determined based on the pixel value and brightness of each pixel point and the positional relationship between each pixel point and the target region in the image.

[0132] In step S506, for each image, the pixel value of each pixel point is corrected based on the weight value corresponding to each pixel point in the image to obtain the corrected pixel value corresponding to each pixel point.

[0133] In step S507, the corrected image corresponding to the image is obtained based on the corrected pixel value corresponding to each pixel point.

[0134] In step S508, the corrected pixel values corresponding to the same pixel points at the same positions in the corrected images are fused to obtain the target pixel value corresponding to the pixel points.

[0135] In step S509, the target image corresponding to the picture is obtained based on the target pixel value corresponding to each pixel point.

[0136] In step S510, the target image is input into the fault detection and diagnosis model corresponding to the power transmission line to obtain the fault detection and diagnosis result of the power transmission line.

[0137] In this embodiment, first, the server can perform target detection processing corresponding to the moving target on the image through the moving target detection model, so as to determine the target region in which the moving target exists in the image, and facilitate subsequent processing of the moving target to eliminate the ghost generated in the fusion process of the moving target. Second, the server comprehensively measures the correlation degree between each pixel point and the moving target based on the pixel value, brightness and positional relationship between the pixel point and the target region, so as to assign a corresponding weight value to each pixel point. Third, the server can eliminate the ghost generated in the fusion process of the moving target in advance before image fusion by correcting the pixel value of each pixel point through the weight value corresponding to the pixel point, reduce the influence of the moving target on the image quality, and further enable the server to obtain a target image with high information content and without ghost. Fourth, the server can realize the fusion processing of multiple corrected images under the condition that the ghost generated in the fusion process of the moving target is eliminated in advance by fusing the corrected pixel values corresponding to the same pixel points in the corrected images through the fusion weight value corresponding to each corrected image, and further obtain a target image with high information content and without ghost, thereby improving the reliability of the fault detection and diagnosis of the power transmission line.

[0138] In order to more clearly illustrate the image processing method provided by the embodiments of the present application, the image processing method will be specifically described below with one specific embodiment, but it should be understood that the embodiments of the present application are not limited thereto. In an exemplary embodiment, the present application also provides a ghost detection and removal method based on an improved Faster R-CNN, specifically comprising the following steps:

[0139] Step 1: feature extraction.

[0140] The server first extracts features from multiple images under different exposure times. Based on multiple images under different exposure times, the details of the same picture under different brightness levels can be captured to improve the dynamic range of the image and make the details clearer. The server uses a convolutional residual network ResNet101 as a feature extraction network to extract features of the image. The deep structure of the convolutional residual network ResNet101 allows the network to learn richer and more abstract image features and high-level semantic information of the image, so as to better capture the details in the image and understand the shape, texture of the objects in the image and their relationship. It has no fixed requirement for the resolution of the input image.

[0141] Step 2: ghost detection.

[0142] The server uses a ghost detection method based on Faster R-CNN; wherein the region generation network is responsible for generating potential candidate boxes containing moving targets, and assigning a confidence score to each candidate box, which represents the likelihood of the candidate box containing a moving target. These candidate boxes are filtered by non-maximum suppression to leave the regions most likely to contain the target. Specifically, all candidate boxes are arranged in descending order of score, and the candidate box with the highest score is added to the final set. For the remaining candidate boxes, calculate their intersection over union with the selected candidate boxes. If the overlap of a candidate box with the selected candidate boxes is greater than a certain threshold of 0.7, the candidate box is removed. Finally, at least one candidate box with a high score and no overlap is obtained. Then, the server uses the finally generated candidate box to detect ghosts, and determines the final target class and target box by applying a classifier and a regressor on each candidate box.

[0143] Step 3: ghost removal.

[0144] The server uses a weight map reflecting the correlation between each pixel point and the moving target to remove ghosts, aiming to assign weights to different pixel points to reduce or eliminate the ghost effect caused by the moving target. Specifically, for each image, a corresponding weight map is generated, and the pixel values of each pixel point of the image are corrected according to the weight map of the image to obtain a corrected image corresponding to the image. Then, multiple corrected images are fused to obtain a target image.

[0145] In this embodiment, the server proposes a ghost detection and removal method based on an improved Faster R-CNN, improves the detection rate of moving targets in images under different exposure times, and further improves the detection rate of ghosts. After detecting the ghost, the ghost is removed by using the weight map, which can effectively solve the problem of ghost detection and removal in high dynamic range imaging technology. The introduction of this technology in the visual automatic monitoring of the power industry can more accurately analyze the monitoring data, reduce the demand for manual intervention, help to establish a more reliable monitoring system, and improve the automation level of the system.

[0146] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0147] Based on the same inventive concept, the embodiments of the present application also provide an image processing device for implementing the above-mentioned image processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations of the image processing method in the above text, which will not be repeated here.

[0148] In one exemplary embodiment, as shown in Figure 6 An image processing device is provided, including: an image acquisition module 602, a target detection module 604, a weight determination module 606, an image correction module 608, and an image fusion module 610, wherein:

[0149] The image acquisition module 602 is configured to acquire multiple images of a picture under different exposure times; each image corresponds to an exposure time.

[0150] The target detection module 604 is configured to determine a target region in which a moving target exists in each image based on a moving target detection model.

[0151] The weight value determination module 606 is configured to determine, according to each image and the corresponding target region, a weight value corresponding to each pixel point in each image, where the weight value is used to represent a correlation degree between the pixel point and the moving target.

[0152] The image correction module 608 is configured to correct each image based on the weight value corresponding to each pixel point in the image, to obtain a corrected image corresponding to each image.

[0153] The image fusion module 610 is configured to fuse the corrected images corresponding to the images to obtain the target image corresponding to the picture.

[0154] In an example embodiment, the weight value determination module 606 is further configured to determine, for each image, a pixel value and a brightness of each pixel point in the image, and a positional relationship between each pixel point and the target region in the image; and determine the weight value corresponding to each pixel point based on the pixel value and the brightness of each pixel point, and the positional relationship between each pixel point and the target region in the image.

[0155] In an example embodiment, the image correction module 608 is further configured to correct, for each image, a pixel value of each pixel point based on the weight value corresponding to each pixel point in the image, to obtain a corrected pixel value corresponding to each pixel point; and obtain the corrected image corresponding to the image based on the corrected pixel value corresponding to each pixel point.

[0156] In an example embodiment, the resolutions of the multiple images are the same, and the resolutions of the corrected images corresponding to the multiple images are the same.

[0157] The image fusion module 610 is further configured to perform fusion processing on the corrected pixel values corresponding to the same pixel point in the corrected images to obtain a target pixel value corresponding to the pixel point, where the same pixel point is a pixel point having the same position in the corrected images; and obtain the target image corresponding to the picture based on the target pixel value corresponding to each pixel point.

[0158] In an example embodiment, the target detection module 604 is further configured to input, for each image, the image into a moving target detection model, to determine multiple candidate regions associated with the moving target in the image and confidences of the multiple candidate regions, where the confidence is used to represent a possibility that the candidate region contains the moving target; and select, based on the confidence of each candidate region, a target region containing the moving target from the multiple candidate regions, and determine a position of the target region in the image.

[0159] In an example embodiment, the picture is a picture centered on a power transmission line.

[0160] The image processing apparatus further includes a fault detection and diagnosis module configured to input the target image into a fault detection and diagnosis model corresponding to the power transmission line to obtain a fault detection and diagnosis result of the power transmission line.

[0161] The modules in the image processing apparatus can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be invoked and executed by the processor to perform operations corresponding to the modules.

[0162] In an example embodiment, a computer device is provided, which can be a server. An internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store image data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement an image processing method.

[0163] Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In an example embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0165] In an example embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0166] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0167] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0168] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0169] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire multiple images of a power transmission line at different exposure times; each image corresponds to one exposure time. Based on the moving target detection model, the target region containing moving targets in each image is determined; For each image, the pixel value and brightness of each pixel in the image are determined, as well as the positional relationship between each pixel and the target region in the image. Based on the pixel value and brightness of each pixel, and the positional relationship between each pixel and the target region in the image, a weight corresponding to each pixel is determined. The weight is used to characterize the degree of association between the pixel and the moving target, and the weight is positively correlated with the degree of association. For each image, the pixel value of each pixel is corrected by using the weight corresponding to each pixel in the image as the pixel value correction coefficient, and the corrected pixel value is obtained. Based on the corrected pixel value, the corrected image is obtained. The corrected images corresponding to each image are fused to obtain the target image corresponding to the scene; the target image is used to determine the fault detection and diagnosis results of the transmission line.

2. The method according to claim 1, characterized in that, The multiple images have the same resolution, and the corresponding corrected images of the multiple images have the same resolution; The process of fusing the corrected images corresponding to each image to obtain the target image corresponding to the scene includes: The corrected pixel values ​​corresponding to the same pixel in each corrected image are fused to obtain the target pixel value corresponding to the pixel; the same pixel refers to the pixel with the same position in each corrected image. Based on the target pixel value corresponding to each pixel, the target image corresponding to the scene is obtained.

3. The method according to claim 1, characterized in that, The moving target detection model determines the target region containing a moving target in each image, including: For each image, the image is input into a moving target detection model to determine multiple candidate regions in the image associated with a moving target, and the confidence level of the multiple candidate regions; the confidence level is used to characterize the probability that the moving target exists in the candidate region; Based on the confidence level of each candidate region, a target region containing the moving target is selected from the plurality of candidate regions, and the position of the target region in the image is determined.

4. The method according to any one of claims 1 to 3, characterized in that, After fusing the corrected images corresponding to each image to obtain the target image corresponding to the scene, the process further includes: The target image is input into the fault detection and diagnosis model corresponding to the transmission line to obtain the fault detection and diagnosis results of the transmission line.

5. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire multiple images of the power transmission line at different exposure times; each image corresponds to one exposure time. The target detection module is used to determine the target regions in each image where there are moving targets, based on the moving target detection model; The weight determination module is used to determine the pixel value and brightness of each pixel in each image, and to determine the positional relationship between each pixel and a target region in the image. Based on the pixel value and brightness of each pixel, and the positional relationship between each pixel and the target region in the image, the module determines the weight corresponding to each pixel. The weight is used to characterize the degree of association between the pixel and the moving target, and the weight is positively correlated with the degree of association. The image correction module is used to correct the pixel value of each pixel for each image by using the weight corresponding to each pixel in the image as the pixel value correction coefficient, to obtain the corrected pixel value corresponding to each pixel, and to obtain the corrected image corresponding to the image based on the corrected pixel value corresponding to each pixel. An image fusion module is used to fuse the corrected images corresponding to each image to obtain the target image corresponding to the scene; the target image is used to determine the fault detection and diagnosis results of the transmission line.

6. The apparatus according to claim 5, characterized in that, The multiple images have the same resolution, and the corresponding corrected images of the multiple images have the same resolution; The image fusion module is further configured to fuse the corrected pixel values ​​corresponding to the same pixel in each corrected image to obtain the target pixel value corresponding to the pixel; the same pixel refers to the pixel with the same position in each corrected image. Based on the target pixel value corresponding to each pixel, the target image corresponding to the scene is obtained.

7. The apparatus according to claim 5, characterized in that, The target detection module is further configured to input the image into the moving target detection model for each image, determine multiple candidate regions in the image associated with the moving target, and the confidence level of the multiple candidate regions; The confidence level is used to characterize the probability that the moving target exists in the candidate region; Based on the confidence level of each candidate region, a target region containing the moving target is selected from the plurality of candidate regions, and the position of the target region in the image is determined.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images