Image processing method and device and related equipment

Abstract

The invention relates to the field of power system field monitoring, in particular to an image processing method and device and related equipment. The method is applied to inspection equipment in an electric power place, and comprises the following steps: obtaining a first image and a second image to be registered and fused; performing at least one round of registration fusion on the first image and the second image until a current registration fusion image obtained based on the nth round of registration fusion process meets a preset standard; wherein any round of registration fusion process comprises the following steps of: intercepting a current local image corresponding to the second image from the first image; and according to the current local image, performing registration fusion on the first image and the second image to obtain the current registration fusion image. The current local image in the first image and the second image are used for registration fusion, a small amount of image local information is effectively used to improve matching efficiency, information interference outside the current local image can be reduced, and registration fusion accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of on-site monitoring of power systems, and in particular to an image processing method, device and related equipment. Background Art

[0002] In the power system, inspection equipment needs to be set up at the site of the power generation system, transmission system, and distribution system to collect on-site images. The on-site images are subsequently used to predict faults in the power system, such as inputting the on-site images into the fault model for prediction.

[0003] According to the different types of cameras in the inspection equipment, the types of scene images include at least infrared images and visible light images; in order to improve the accuracy and efficiency of fault prediction, different types of scene images that can reflect different scene characteristics can be registered and fused. In the prior art, the process of registering and fusing two scene images obtained by the inspection equipment usually includes: determining the coordinate transformation matrix between the coordinate systems of the two scene images according to the two scene images, and then fusing the two scene images based on the coordinate transformation matrix to obtain a registered fused image.

[0004] In the actual image registration process, the accuracy requirement for the coordinate transformation matrix is ​​very high. Due to the low clarity of the on-site images or the large overlapping interference of power equipment in the images, the calculation error of the coordinate transformation matrix is ​​large, making it difficult to effectively perform image registration and fusion, which in turn leads to a decrease in the accuracy of the fault prediction results based on the registered and fused images, ultimately affecting the safe operation of the power system. Summary of the invention

[0005] The purpose of the present application is to provide an image processing method, apparatus and related equipment to solve the technical problems of low calculation efficiency of the coordinate transformation matrix between the first image and the second image and large errors introduced in the existing image registration and fusion solutions, resulting in large distortion of the registered and fused images.

[0006] In a first aspect, the present application provides an image processing method, which is applied to inspection equipment in power sites, and the method comprises:

[0007] Acquire a first image and a second image to be registered and fused;

[0008] Performing at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard;

[0009] The registration and fusion process of any round includes:

[0010] intercepting a current local image corresponding to the second image in the first image; wherein the current local image contains part or all of the image content in the second image; and the current local images intercepted in different rounds of the registration and fusion process are different;

[0011] According to the current local image, the first image and the second image are registered and fused to obtain the current registered and fused image.

[0012] In a second aspect, the present application provides an image processing device applied to inspection equipment in power sites, the device comprising: an image acquisition module and an image registration and fusion module;

[0013] The image acquisition module is used to acquire the first image and the second image to be registered and fused;

[0014] The image registration and fusion module is used to perform at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard;

[0015] The registration and fusion process of any round includes:

[0016] intercepting a current local image corresponding to the second image in the first image; wherein the current local image contains part or all of the image content in the second image; and the current local images intercepted in different rounds of the registration and fusion process are different;

[0017] According to the current local image, the first image and the second image are registered and fused to obtain the current registered and fused image.

[0018] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store software programs, and the processor runs or executes the software programs stored in the memory so that the electronic device implements the above-mentioned image processing method.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes executed by a processor, wherein the program codes include codes for implementing the above-mentioned image processing method.

[0020] In a fifth aspect, the present application provides a computer program product, including a program code, which enables a computer to execute the above-mentioned image processing method when the computer runs the computer program product.

[0021] Beneficial effects:

[0022] The present application provides an image processing method, the method comprising: acquiring a first image and a second image to be registered and fused; performing at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard; wherein any round of registration and fusion process comprises: intercepting a current local image corresponding to the second image in the first image; wherein the current local image contains part or all of the image content in the second image; the current local images intercepted in different rounds of registration and fusion process are different;

[0023] In summary, the present application uses the current local image corresponding to the second image in the first image to perform registration and fusion on the first image and the second image. Since the image content in the current local image corresponds to the image content in the second image, compared with using the entire first image and the second image for registration and fusion, the present application uses the current local image to perform registration and fusion on the first image and the second image, which can effectively improve the efficiency of registration and fusion, and avoid the interference of cluttered scenes in the image, thereby improving the accuracy of registration and fusion;

[0024] Secondly, the present application also provides a registration and fusion result verification step for the current registration and fusion image and a cyclic registration and fusion method based on the registration and fusion result verification step, so that the inspection equipment in the power system can immediately determine whether the current registration and fusion image based on the first image and the second image is distorted. If distortion occurs, the next round of registration and fusion process is immediately performed on the first image and the second image until the obtained registration and fusion image meets the preset standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0027] Figure 2 A logical schematic diagram of an image processing method provided in an embodiment of the present application;

[0028] Figure 3 An example diagram of the current registered fusion image provided in the embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The power system can provide a stable and reliable power supply for social production and life. The power system mainly includes: power generation system, transmission system and distribution system. The interruption or instability of power generation in the power system will lead to stagnation of industrial production, obstruction of commercial activities and inconvenience in residents' lives; power system failures may cause safety accidents such as fire and explosion, posing a threat to the safety of life and property. Therefore, ensuring the safe operation of the power system is of great significance to social stability, economic development and public safety.

[0031] In the prior art, inspection equipment is usually set up on-site in the power system to collect on-site images, which can then be pre-processed and input into the machine learning model for fault prediction. Among them, the inspection equipment includes: drones moving in the air and inspection robots moving on the ground; the specific content of the on-site images varies depending on the on-site settings of the power system and the distribution of the inspection equipment. In many cases, a single on-site image may display multiple devices of the power system.

[0032] In practical applications, different types of on-site images can present different characteristic information. For example, visible light images have a higher resolution and can clearly present abnormalities in the appearance of equipment, such as equipment damage, oil leakage, exposed lines, etc., while infrared images can present the surface temperature distribution of equipment.

[0033] In actual implementation, the visible light image and infrared image that can present different feature information can be registered and fused to obtain a registered fused image. Compared with fault prediction based on visible light images and infrared images respectively, since the registered fused image can present a variety of feature information, using the registered fused image for fault prediction can obviously improve the efficiency of fault prediction.

[0034] In the prior art, the main idea of ​​the visible light image and infrared image registration and fusion scheme is to determine the coordinate transformation matrix between the coordinate system of the visible light image and the coordinate system of the infrared image based on the visible light image and the infrared image, and then fuse the visible light image and the infrared image based on the coordinate transformation matrix to obtain a registered fused image, wherein image registration is a key step in obtaining a fused image. The purpose of image registration is to superimpose multiple images of different resolutions and different perspectives in the same scene, fuse the thermal detail information of the infrared image with the texture information of the visible light image, and thus present the texture and temperature distribution information of the device in a single image.

[0035] In practical applications, the steps of the visible light image and infrared image registration and fusion scheme include: image preprocessing, feature point detection, contour angle direction calculation, coarse registration stage, fine registration stage and fusion processing; among them, the contour angle direction calculation involves steps such as feature point selection and contour angle estimation. Due to the large errors in feature point selection and contour angle estimation, the coordinate transformation matrix obtained in the fine registration stage is biased, making it difficult to effectively perform the registration and fusion of visible light and infrared multimodal images. The accuracy of fault prediction based on this will also be sharply reduced, making it impossible to effectively monitor the power equipment.

[0036] In order to solve the above problems, the present application proposes an image processing solution, which is applied to inspection equipment in power sites. The image processing solution proposed in the present application is now explained in conjunction with the drawings in the specification.

[0037] First, the present application embodiment proposes an image processing method, such as Figure 1 As shown, Figure 1 The following is a flow chart of an image processing method provided in an embodiment of the present application. The method includes: S100 to S200, and the details are as follows:

[0038] S100: Acquire a first image and a second image to be registered and fused.

[0039] Specifically, the inspection equipment at the power site uses different cameras to simultaneously photograph the site of the power site to obtain a first image and a second image.

[0040] In the embodiment of the present application, the first image and the second image are respectively a visible light image and an infrared image; wherein, for a general dual-spectrum camera, the size of the visible light image is larger than the size of the infrared image and the image content displayed by the visible light image is more than the image content displayed by the infrared image.

[0041] In an embodiment of the present application, image content refers to a photographed object presented in an image taken at a certain shooting location, at a certain shooting angle and at a certain shooting distance. The photographed object may be a power site and one or more power equipment arranged in the power site. In actual operation, due to the limited shooting range of the shooting equipment, the image content usually only includes part of the power site and part of the power equipment arranged in the part of the power site. When the power equipment captured in the image is located at the edge of the image, only a part of the power equipment may be captured. For example, if a dual-spectrum camera is used to shoot a visible light image and an infrared image of power equipment A, the image content displayed by the visible light image may include power equipment A and power equipment B adjacent to power equipment A, but the infrared image may only display power equipment A.

[0042] In practical applications, since the size of a visible light image is usually larger than that of an infrared image, the image content of the visible light image and the infrared image are not exactly the same even if they are taken at the same shooting location, at the same shooting angle, and at the same shooting distance. However, there is generally an image area in the visible light image that corresponds to the image content of the infrared image, that is, the visible light image and the infrared image taken at the same shooting location, at the same shooting angle, and at the same shooting distance have "corresponding image content". The difference is that the image content of the visible light image presents the object being photographed under natural light, and the image content of the infrared image presents the object being photographed under infrared light. The visible light image and the infrared image can provide different image feature information when presenting image content.

[0043] In an embodiment of the present application, image feature information refers to specific information about image features of an image region where image content is located in an image. Image features may be color distribution, contour, texture, color depth, etc. For example, image feature information including color distribution data, contour data, and texture data of an image region where image content is located may be determined in a visible light image. For another example, image feature information including color depth data reflecting thermal target characteristics may be determined in an infrared image where image content is located.

[0044] In actual operation, when there is any difference in the shooting location, shooting angle or shooting distance when the two images are taken, the visible light image and the infrared image may not have the "corresponding image content" mentioned above. Therefore, in order to ensure that the infrared image and the visible light image have "corresponding image content", it is necessary to ensure that the infrared image and the visible light image are taken at the same shooting location, at the same shooting angle and at the same shooting distance.

[0045] The technical solution of the present application will be explained in detail later by taking the first image being a visible light image and the second image being an infrared image as an example.

[0046] S200: performing at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard.

[0047] Specifically, the existing registration and fusion scheme for visible light images and infrared images with “corresponding image contents” only has a simple registration and fusion process, which results in the inability to automatically perform feedback correction when the registered and fused images are distorted.

[0048] In order to solve the above problems, the present application adds a distortion judgment process to the registration and fusion scheme, which is used to judge whether the registered and fused image meets the preset standards, that is, whether distortion occurs; the present application also sets the registration and fusion process to be performed for at least one round, because if the result obtained by the first round of the registration and fusion process meets the non-distortion condition, it is considered that no distortion occurs, and the current registration and fusion process of the visible light image and the infrared image is terminated.

[0049] In actual implementation, the registration fusion process of any round includes: S210 to S230, the details of which are as follows:

[0050] S210: intercepting a current partial image corresponding to the second image in the first image;

[0051] The current local image includes part or all of the image content in the second image; and the current local images intercepted in different rounds of registration and fusion processes are different.

[0052] Specifically, in the embodiment of the present application, first, according to the current partial image A of the first image n The coordinate transformation matrix of the first image and the second image is calculated based on the second image, rather than directly calculating the coordinate transformation matrix based on the entire first image and the second image, to avoid redundant calculation process of features and introduction of too much noise when calculating based on the entire first image. Therefore, in the embodiment of the present application, try to first select the current local image containing less image content in the second image to jointly determine the coordinate transformation matrix with the second image, and then perform registration and fusion processing on the first image and the second image according to the coordinate transformation matrix to obtain a registered fused image; if the registered fused image obtained based on the current local image containing less image content in the second image does not meet the preset standard, the current local image containing more image content (which can be all the image content) can be selected to jointly determine the coordinate transformation relationship with the second image.

[0053] In actual operation, since the current local image A n Compared with the first image, the current local image A has a smaller image area. n The feature calculation between the first image and the second image is more targeted, which can reduce redundant feature calculations and further improve the calculation efficiency of the coordinate transformation matrix.

[0054] In an embodiment of the present application, the current local image captured from the first image in different rounds of registration and fusion processes is different, and the image is equivalent to an image expanded outward from the current local image captured in the previous round of registration process as the center; wherein, the current local image in the first registration and fusion process can be obtained by randomly capturing the first image.

[0055] For example, if the current local image A captured in the nth round of registration and fusion process n The current registered fusion image B obtained by registration and fusion n If the preset standard is not met, the (n+1)th round of registration and fusion process is started; in the (n+1)th round of registration and fusion process, the current local image A corresponding to the nth round is used in the visible light image. n The image is expanded to the left, right, top and bottom of the area, and then the expanded image area is intercepted to obtain the current local image A of the (n+1)th round. n+1 .

[0056] If based on the current local image A n+1 The current registered fusion image B obtained by registration and fusion n+1 If it still does not meet the preset standard, the (n+2)th round of registration and fusion process is started; in the (n+2)th round of registration and fusion process, the current local image A corresponding to the (n+1)th round is used in the visible light image. n+1 The registration calculation is stopped after the registration process is completed and no valid result can be obtained.

[0057] It should be emphasized that the position and area of ​​the screenshot area added in each round can be determined according to actual needs, and this application does not make any specific limitations on this.

[0058] In actual operation, the coordinate conversion matrix between the two images is determined by calculating the image coordinates of each pixel position in the two images. Therefore, as the size of the current local image continues to increase, the image feature information contained in the current local image will gradually increase. Based on this, the coordinate conversion matrix is ​​accurately determined to avoid the image coordinates of more pixel positions required due to the local image size being too large, resulting in redundant calculations. For example, assuming that the image content of the second image, i.e., the infrared image, is A, and the local image corresponding to the image content A in the image content of the first image, i.e., the visible light image, is B, and assuming that when the local image C in the visible light image includes the image content of the local image B and the image area occupies 3 / 5 of the visible light image, the accurate coordinate conversion matrix can be determined with the infrared image, if the local image D in the visible light image includes the image content of the local image B and the image area occupies 4 / 5 of the visible light image is directly used to calculate the coordinate conversion matrix with the infrared image, it will result in the need to calculate the image coordinates corresponding to the local image of 1 / 5 of the visible light image in the process of calculating the coordinate conversion matrix, resulting in redundant calculations.

[0059] In actual operation, since each pair of visible light images and infrared images with "corresponding image contents" photographs different objects, it is not clear before calculating the coordinate transformation matrix which local image in the visible light image can be used to calculate the coordinate transformation matrix (i.e. the local image C mentioned above). Therefore, it is necessary to confirm step by step to find the local image that can be used to calculate the coordinate transformation matrix; "just" means that the accurate coordinate transformation matrix can be calculated but there is no need to calculate too many image coordinates in the process.

[0060] In one implementation, when n=1, S210 includes: Step (1) to Step (3), the details of which are as follows:

[0061] Step (1): based on the second image, intercepting m candidate partial images in the first image;

[0062] Wherein, m is a positive integer; the size of the candidate partial image is smaller than or equal to the size of the second image.

[0063] Specifically, in the embodiment of the present application, the objects of registration and fusion are the current local image and the second image in the first image; in actual operation, in order to improve the accuracy and efficiency of determining the "current local image", the "current local image" is determined by technical means; in the process of determining the current local image, it is first necessary to randomly intercept multiple candidate local images in the first image, the size of the candidate local images may be less than or equal to the size of the infrared image, and the candidate local images may include part or all of the image content of the second image; subsequently, the candidate local image most similar to the second image is determined from the multiple candidate local images, and then the candidate local image is determined as the "current local image" for image registration with the second image;

[0064] Among them, the purpose of selecting the "most similar" alternative local image from multiple alternative local images is to ensure that the current local image and the second image have corresponding image content, that is, to ensure that the current local image and the second image are images acquired for the same object / part; it should be emphasized that the current local image and the second image mentioned here have corresponding image content.

[0065] In actual operation, the image content of the current local image may be more than that of the second image, or the image content of the current local image may be less than that of the second image; in the first round of registration and fusion of the visible light image and the infrared image, the image content of the current local image A1 determined by the candidate local image is usually less than or equal to the image content of the second image; in the nth round of registration and fusion of the visible light image and the infrared image, the image content of the current local image A1 determined by the candidate local image is usually less than or equal to the image content of the second image; n-1 The current local image A of the nth round determined nThe image content of the first image may be greater than the image content of the second image.

[0066] Step (2): confirm the similarity between each candidate partial image in the m candidate partial images and the second image, and obtain m similarities.

[0067] Step (3): The candidate local image corresponding to the similarity with the largest value among the m similarities is determined as the current local image of the current round.

[0068] Specifically, the candidate partial image with the greatest similarity is confirmed as the current partial image A1 of the current round, that is, the image contents contained in the current partial image A1 and the second image are closest, and the number of corresponding matching feature points calculated in the current partial image A1 and the second image is the largest.

[0069] In one implementation, when n≥2, S210 includes: step (4), the details of which are as follows:

[0070] Step (4): based on the current local image of the (n-1)th round intercepted in the registration and fusion process of the (n-1)th round, intercept the current local image of the nth round in the first image;

[0071] The current local image of the nth round includes the current local image of the (n-1)th round, and the size of the current local image of the nth round is larger than the size of the current local image of the (n-1)th round.

[0072] Specifically, in actual implementation, when registering and fusing the visible light image and the infrared image, the current registered fusion image B obtained by the registration fusion is n The preset criteria may not be met, so the registration fusion process may be performed in multiple rounds.

[0073] In this implementation, the current local image A of the nth round is captured during the nth round of registration and fusion process. n is the current local image A of the (n-1)th round obtained during the (n-1)th round of registration and fusion process n-1 Determined; in actual implementation, according to the current local image A of the (n-1)th round n-1 The size and A n-1 At the image position in the visible light image, a larger current local image A is intercepted in the visible light image and contains the (n-1)th round. n-1 The current local image A of the nth round n .

[0074] In one implementation, the current local image of the nth round obtained during the nth round of registration and fusion is determined by performing horizontal and vertical expansion processing in the first image with the current local image of the (n-1)th round as the center.

[0075] Specifically, according to Figure 2 The left side of the figure shows the first round of registration and fusion process. Figure 2 As can be seen from the “first image” of the “nth round of registration and fusion process” on the right, there is a box displayed in the “first image” on the left, which indicates the image position of the current local image A1 of the first round obtained by cutting the first image in the first image; there are two boxes displayed in the “first image” on the right, the smaller first box indicates the current local image A1 of the (n-1)th round. n-1 , the larger second box indicates the current local image A of the nth round n ;

[0076] In the implementation of this application, the current local image A of the nth round n , is the current local image A of the (n-1)th round n-1 The expansion distance between the first box and the second box is the step length. The step length can be determined according to actual needs, and this application does not make any specific limitation on this.

[0077] S220: According to the current local image, register and fuse the first image and the second image to obtain a current registered and fused image.

[0078] Specifically, after determining the coordinate transformation matrix according to the current local image, the first image and the second image are fused based on the coordinate transformation matrix to obtain the current registered fused image.

[0079] The area corresponding to the second image in the current registered fusion image can reflect both the original image feature information of the second image and the image feature information of the first image.

[0080] In one implementation, Figure 2 As shown, Figure 2 The logical diagram of the image processing method provided in the embodiment of the present application, S220 includes: Step (5) to Step (7), the details of which are as follows:

[0081] Step (5): According to the image position of the current partial image of the current round in the first image, the image area of ​​the first image excluding the current partial image is subjected to occlusion processing to obtain a current occluded image.

[0082] Specifically, according to Figure 2 It can be seen that the current occlusion image K nContains the current local image A n , the current occlusion image K n Contains the current local image A n and keep the current local image A n The original pixels of the remaining pixels are 0; the current occlusion image K n This is equivalent to only displaying the current local image A. n The image feature information of the first image is not displayed except for the current local image A. n The first image of the image feature information of other image areas outside the current local image A; in actual operation, the "other image areas" may be displayed as black areas; n In the current occlusion image K n The position of is the same as that in the first image, and then in K n The coordinates of the feature points collected in the image are the same as the coordinates of the feature points in the first image.

[0083] Step (6): Determine the coordinate transformation matrix between the coordinate system where the first image is located and the coordinate system where the second image is located based on the current occluded image and the second image.

[0084] Specifically, the steps of determining the coordinate transformation matrix include: Step (6.1) to Step (6.5), the details of which are as follows:

[0085] Step (6.1): Image processing: For the current occluded image K n Perform grayscale transformation processing on the infrared image I;

[0086] Specifically, the current occlusion image K n The size is the same as the first image; in actual operation, for the current occluded image K n The grayscale transformation is performed on the infrared image I and the grayscale level is scaled to [0,255]. In addition, the current occlusion image K is transformed according to the vertical resolution of the infrared image I. n The vertical resolution of the infrared image I is adjusted to be the same as the vertical resolution of the infrared image I; the horizontal resolution is adjusted according to the image ratio of the first image;

[0087] Step (6.2): ​​Feature point detection: Use the Canny edge detection algorithm to determine the current occluded image K n and the image edge map of the infrared image I; wherein the image edge map is the current occlusion image K n The image after the edge extraction from the infrared image I. The edge presented in the image edge map is usually the corresponding current occlusion image K n And the place where the gray value in the infrared image I changes sharply.

[0088] By traversing the image edge map based on a local-global curvature scale space detector (CSS), each contour line in the image edge map is determined to obtain a contour line set S;

[0089] The definition of the contour set S is as follows:

[0090]

[0091] Among them, Γ j Represents the jth contour line in the contour set; 1≤j≤N s ;

[0092] n represents the contour line Γ j The number of all feature points on ;

[0093] Step (6.3): Calculation of contour angle direction: When the feature points in the assumed contour set S are and feature points For feature points At the closest point of the local minimum curvature on the contour lines [1,f] and [f,n], three feature points can form a feature triangle;

[0094] The feature points The corresponding angle bisector vector The direction of the feature point The main direction of the image is to achieve invariance to image rotation and perspective difference;

[0095] Step (6.4): Rough registration stage: Based on the premise that the closer the theoretical rotation angle is to the actual rotation angle, the more registered feature points there are, a histogram is drawn with the theoretical rotation angle as the horizontal axis and the number of correctly registered feature points as the vertical axis;

[0096] The parabolic interpolation method is used on the histogram to estimate the rotation angle. When the registration is correct, the connecting lines between the feature points are theoretically of the same length and parallel to each other. The rotation angle estimation and connecting line consistency theory are used for rough registration to filter out false registration points with large errors.

[0097] Step (6.5): Fine registration stage: Through fine registration based on global iterative optimization, the position deviation of feature points is further reduced; assuming P c1 and P c2 is the final correct matching point, M is the coordinate transformation matrix, and the minimum square loss can be obtained:

[0098] minf=‖P c1 MP c2 ‖2;

[0099] The coordinate transformation matrix M is as follows:

[0100]

[0101] Then P c2 The corresponding fit points are as follows:

[0102]

[0103] The residuals of the two are expressed as follows:

[0104] ε(i)=||P 2fit (i)-P c2 (i)||2,i=1,…,N c ;

[0105] Will Sort in ascending order, c2 The points corresponding to the last 1 / 4 elements in are replaced by the fitting points P c2 , and then continuously update P by repeating steps (6.2) to (6.5) c2 , until the residual is zero, thus completing the global iterative optimization and achieving the purpose of fine registration.

[0106] Step (7): According to the coordinate transformation matrix, the first image and the second image are fused to obtain a current registered fused image.

[0107] Specifically, after the coordinate conversion matrix is ​​determined, the visible light image and the infrared image are fused to obtain a current registered fused image, which is a result of the second image being transformed in coordinates and superimposed on the first image.

[0108] In the embodiment of the present application, the size of the infrared image is smaller than that of the visible light image, and the first area in the current registered fused image refers to an image area having both image feature information of the visible light image and feature information of the infrared image.

[0109] S230: Determine whether the current registered and fused image meets the preset standard; if so, the registration and fusion ends; if not, repeat step S210.

[0110] Specifically, the purpose of judging whether the current registered fusion image meets the preset standard is to determine whether the current registered fusion image is distorted. If no distortion occurs, the registration and fusion process of the current visible light image and infrared image is terminated. If distortion occurs, it means that the feature points collected in the current local image are not sufficient to accurately determine the coordinate transformation matrix, so S210 needs to be repeated.

[0111] In one implementation, the current registered fused image includes a first region, and the first region is a region of the current registered fused image corresponding to the current local image of the current round; S230 includes: steps (8) to (9), the details of which are as follows:

[0112] Step (8): Determine the current boundary shape of the first region in the current registered fused image.

[0113] Step (9): Determine whether the current registered fusion image meets the preset standard based on the relationship between the absolute value of the difference between each inner angle of at least one inner angle of the current boundary figure and 90° and the corresponding preset threshold.

[0114] Specifically, in the embodiment of the present application, the absolute value of the difference between the angle of the current boundary figure of the first area and 90° is used to measure the degree of distortion of the current registered fused image.

[0115] like Figure 3 As shown, Figure 3 This is an example diagram of the current registered fusion image provided in the embodiment of the present application. Figure 3 In the figure, the current boundary figure of the first area is set to a parallelogram, and the corresponding internal angle may be an acute angle or an obtuse angle. Since the parallelogram is a regular polygon and any two adjacent internal angles in the parallelogram are complementary, it is usually only necessary to judge the size relationship between the absolute value of the difference between one of the internal angles and 90° and its corresponding preset threshold; when the absolute value is less than the preset threshold, it can be determined that the current registered fusion image meets the preset standard; wherein, the setting of the preset threshold can be determined according to actual needs, and the present application does not make specific limitations on this.

[0116] In actual operation, the current boundary graphic may be an irregular polygon; if the current boundary graphic is an irregular polygon, multiple inner angles may be selected according to actual needs to determine whether the current registered fusion image meets the preset standards. Accordingly, when multiple inner angles need to be selected to determine whether the current registered fusion image meets the preset standards, the preset thresholds should also be multiple and have a one-to-one correspondence with the multiple inner angles; wherein, the selection of inner angles in the irregular polygon may be determined according to actual needs, and this application does not make any specific limitations on this.

[0117] In summary, the present application optimizes the image registration method in the prior art. First, the coordinate transformation matrix is ​​determined by the current local image and the infrared image. The feature points provided by the current local image all correspond to the pixel points in the infrared image. Compared with using the entire visible light image and the infrared image to determine the coordinate transformation matrix, the efficiency of the coordinate transformation matrix can be effectively improved. Secondly, the present application also provides a distortion detection step for the current registered fusion image and a cyclic registration and fusion method based on the distortion detection step, so that the inspection equipment can instantly determine whether the current registered fusion image is distorted. If distortion occurs, S210 is immediately repeated to re-determine the current registered fusion image.

[0118] Second, the present application provides an image processing device, which is applied to inspection equipment in power places, such as Figure 4 As shown, Figure 4 This is a structural diagram of an image processing device provided in an embodiment of the present application, the device includes: an image acquisition module 310 and an image registration and fusion module 320;

[0119] An image acquisition module 310 is used to acquire a first image and a second image to be registered and fused;

[0120] An image registration and fusion module 320 is used to perform at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard;

[0121] Among them, the registration fusion process of any round includes:

[0122] A current local image corresponding to the second image is intercepted in the first image; wherein the current local image contains part or all of the image content in the second image; and the current local images intercepted in different rounds of registration and fusion processes are different;

[0123] According to the current local image, the first image and the second image are registered and fused to obtain a current registered and fused image.

[0124] In one implementation, when n=1, the image acquisition module 310 is further configured to intercept m candidate partial images in the first image according to the second image; wherein m is a positive integer; and the size of the candidate partial image is less than or equal to the size of the second image;

[0125] The image acquisition module 310 is further used to determine the similarity between each candidate partial image in the m candidate partial images and the second image, and obtain m similarities;

[0126] The image acquisition module 310 is further configured to determine the candidate partial image corresponding to the similarity with the largest value among the m similarities as the current partial image of the current round.

[0127] In one implementation, when n≥2, the image acquisition module 310 is further configured to intercept the current local image of the nth round in the first image according to the current local image of the (n-1)th round intercepted in the registration and fusion process of the (n-1)th round;

[0128] The current local image of the nth round includes the current local image of the (n-1)th round, and the size of the current local image of the nth round is larger than the size of the current local image of the (n-1)th round.

[0129] In one implementation, in the image acquisition module 310, the current local image of the nth round captured during the nth round of registration and fusion is determined by performing horizontal and vertical expansion processing in the first image with the current local image of the (n-1)th round as the center.

[0130] In one implementation, the image registration and fusion module 320 is further configured to perform occlusion processing on an image region of the first image excluding the current local image according to an image position of the current local image of the current round in the first image to obtain a current occluded image;

[0131] The image registration and fusion module 320 is further used to determine a coordinate transformation matrix between a coordinate system where the first image is located and a coordinate system where the second image is located according to the current occluded image and the second image;

[0132] The image registration and fusion module 320 is further configured to perform a fusion process on the first image and the second image according to the coordinate transformation matrix to obtain a current registered and fused image.

[0133] In one implementation, the current registered fused image includes a first region, and the first region is a region in the current registered fused image corresponding to the first image; the image registration and fusion module 320 is further used to determine a current boundary graphic of the first region in the current registered fused image;

[0134] The image registration and fusion module 320 is further used to determine whether the current registered and fused image meets the preset standard based on the size relationship between the absolute value of the difference between each inner angle of at least one inner angle of the current boundary figure and 90° and the corresponding preset threshold.

[0135] Third, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, steps S100 to S200 provided in the above embodiment are implemented.

[0136] Fourthly, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, steps S100 to S200 of the above embodiment are executed.

[0137] Fifth, the present application also provides a computer program product, including a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can refer to steps S100 to S200 of the method embodiment, which will not be repeated here.

[0138] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, mechanical or other forms.

[0139] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0141] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM) random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0142] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0143] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The inspection device is applied to the power site, and the method comprises: Acquire a first image and a second image to be registered and fused; Performing at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard; The registration and fusion process of any round includes: intercepting a current local image corresponding to the second image in the first image; wherein the current local image contains part or all of the image content in the second image; and the current local images intercepted in different rounds of the registration and fusion process are different; According to the current local image, the first image and the second image are registered and fused to obtain the current registered and fused image.

2. The method according to claim 1, characterized in that When n=1, intercepting a current partial image corresponding to the second image in the first image includes: According to the second image, intercepting m candidate partial images in the first image; Wherein, m is a positive integer; the size of the candidate partial image is smaller than or equal to the size of the second image; Determine the similarity between each of the m candidate partial images and the second image to obtain m similarities; The candidate partial image corresponding to the similarity with the largest value among the m similarities is determined as the current partial image of the current round.

3. The method according to claim 2, characterized in that When n≥2, intercepting a current partial image corresponding to the second image in the first image includes: According to the current local image of the (n-1)th round intercepted in the registration and fusion process of the (n-1)th round, intercepting the current local image of the nth round in the first image; The current local image of the nth round includes the current local image of the (n-1)th round, and the size of the current local image of the nth round is larger than the size of the current local image of the (n-1)th round.

4. The method according to claim 3, characterized in that The current local image of the nth round obtained during the registration and fusion process of the nth round is determined by performing horizontal and vertical expansion processing in the first image with the current local image of the (n-1)th round as the center.

5. The method according to claim 2, characterized in that: The performing registration and fusion on the first image and the second image according to the current local image includes: According to the image position of the current partial image of the current round in the first image, an image region of the first image excluding the current partial image is subjected to occlusion processing to obtain a current occluded image; according to the current occluded image and the second image, a coordinate transformation matrix between a coordinate system where the first image is located and a coordinate system where the second image is located is determined; According to the coordinate transformation matrix, a fusion process is performed on the first image and the second image to obtain the current registered fused image.

6. The method according to claim 1, characterized in that The current registered fused image includes a first region, and the first region is a region in the current registered fused image corresponding to the first image; The method further comprises: performing fusion processing on the first image and the second image to obtain the current registered fused image. Determining a current boundary graphic of the first area in the current registered and fused image; Whether the current registered fused image meets the preset standard is determined according to the magnitude relationship between the absolute value of the difference between each of the at least one inner angle of the current boundary figure and 90° and the corresponding preset threshold.

7. An image processing device, characterized in that: The inspection equipment used in power places includes: an image acquisition module and an image registration and fusion module; The image acquisition module is used to acquire the first image and the second image to be registered and fused; The image registration and fusion module is used to perform at least one round of registration and fusion on the first image and the second image until a current registered and fused image obtained based on the nth round of registration and fusion process meets a preset standard; The registration and fusion process of any round includes: intercepting a current local image corresponding to the second image in the first image; wherein the current local image contains part or all of the image content in the second image; and the current local images intercepted in different rounds of the registration and fusion process are different; According to the current local image, the first image and the second image are registered and fused to obtain the current registered and fused image.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store a software program, and the processor runs or executes the software program stored in the memory so that the electronic device implements the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes executed by a processor, wherein the program codes include code for implementing the image processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises program codes, and when a computer runs the computer program product, the computer executes the image processing method according to any one of claims 1 to 6.

Images