Target tracking-based image alignment method for transformer substation scene

Abstract

The invention relates to the field of image processing, and provides a target tracking-based image alignment method for a substation scene. The method aims to solve the problem that the positions of main equipment are inconsistent in multiple shot images, so that the accuracy and efficiency of image processing and analysis are improved. According to the technical scheme, the method comprises the steps that the position of a main device on a to-be-aligned image is obtained through a target tracking algorithm, similarity comparison is conducted on multiple target frames tracked on the to-be-aligned image and the main device on a reference image, and the device position with the highest similarity is reserved. Thirdly, feature point detection is carried out on the reference image and the to-be-aligned image, key points are screened through a manual labeling frame and a target tracking frame, and the key points in a target frame are reserved in a unified mode; and finally, feature point matching, matching point screening and transformation matrix calculation are carried out, affine transformation is carried out on the to-be-aligned image by using the transformation matrix, and the to-be-aligned image is aligned to the reference image. The method is especially suitable for image alignment requirements of main equipment in a transformer substation scene.

Description

Technical Field

[0001] The invention relates to the field of image processing and provides an image alignment method based on target tracking in a substation scene. Background Art

[0002] Image alignment is a classic technology in the field of computer vision and is widely used in the fields of multi-frame image fusion, video anti-shake, and image stitching. For substation scenes, the image content consists of the main equipment and background information, focusing on the equipment and ignoring the background. In the process of multiple image captures, there is a certain deviation in the position of the camera, resulting in inconsistent positions of the main equipment in the multiple images. The problem of inconsistent positions of the main equipment in the multiple images will lead to the following major problems:

[0003] 1. Difficulties in image analysis and processing: In certain scenarios such as substations, if the position of the main equipment in multiple images is inconsistent, it will cause difficulties in subsequent image analysis and processing. For example, when monitoring the status of substation equipment, it is necessary to compare images at different time points. If the equipment position is inconsistent, it is impossible to accurately perform status analysis and change detection.

[0004] 2. Affecting image stitching and quality: In image stitching applications, such as creating panoramas or high-resolution images, multiple images need to be aligned and fused. If the positions of the main devices are inconsistent, the stitched image will be misaligned or blurred, affecting the overall image quality and visual effect.

[0005] 3. Increased need for manual intervention: In order to correct the inconsistency of device position in the image, manual intervention may be required, such as manually adjusting the image alignment. This is not only time-consuming and labor-intensive, but also prone to errors, especially when processing large amounts of image data.

[0006] 4. Reduce the efficiency and reliability of automated systems: In automated image analysis systems, such as automated monitoring and inspection systems, inconsistencies in equipment location can reduce the efficiency and reliability of the system. The system may not be able to accurately identify and track equipment, affecting its performance and decision-making.

[0007] 5. Limiting the application of advanced image processing technology: Some advanced image processing technologies, such as 3D reconstruction and dynamic analysis, rely on precise image alignment. If the device position in the image is inconsistent, the application of these technologies will be limited and cannot reach their full potential.

[0008] Therefore, solving the problem of inconsistent positions of the main device in multiple captured images is crucial to improving the accuracy and efficiency of image processing and analysis. In this case, it is necessary to align the images so that the positions of the main devices are consistent. There are several existing image alignment methods:

[0009] Global alignment: The basic process of global image alignment is usually feature point detection, feature point matching, matching point screening, transformation matrix calculation, and alignment of images according to the transformation matrix. This alignment method is more dependent on the accuracy of selected key points, feature detection and matching. For images with complex backgrounds, if more key points on the background are selected, it is difficult to ensure that the content of the device is aligned.

[0010] Local alignment: Local image alignment is a patch-based matching method that divides the image into multiple grid-like regions, and constructs geometric transformations of multiple regions by matching local features. Each region is differentiated into a plane for alignment, which solves the problem of depth parallax to a certain extent. However, compared to global alignment, local alignment requires calculating the match of each patch, which is computationally intensive, especially for high-resolution videos or dense grid divisions, and there is actually no need to do fine alignment of the background.

[0011] Optical flow method: The optical flow method directly estimates the motion information at the pixel level, that is, the instantaneous motion vector of each pixel from one image to another, which makes it perform well in image alignment tasks with complex motion patterns (such as motion of non-coplanar objects, depth changes, non-rigid motion, etc.). Of course, the optical flow method also has some limitations, such as extreme lighting changes between the reference image and the image to be aligned, large-area occlusion, and the difficulty in processing repeated textures or textureless areas of the image. Summary of the invention

[0012] The present invention aims to solve the problem of inconsistent positions of main devices in multiple captured images, which results in the inability to accurately align and fuse image content in applications such as image stitching, video stabilization or multi-frame image fusion, affecting visual effects and analysis accuracy.

[0013] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0014] The present invention provides an image alignment method based on target tracking in a substation scene, comprising the following steps:

[0015] Step 1: Use a rectangular frame to mark the location of the main equipment in the reference diagram by manual marking;

[0016] Step 2: Use the target tracking algorithm to obtain the position of the main device on the image to be aligned, that is, the frame obtained by target tracking;

[0017] Step 3: There may be other devices similar to the main device on the image to be aligned. In this case, it is necessary to compare the similarity of each tracked result with the main device on the reference image and retain the device position with the highest similarity;

[0018] Step 4: Detect feature points of the reference image and the image to be aligned, use the annotation box to filter the key points of the reference image, and use the target tracking box to filter the key points in the image to be aligned;

[0019] Step 5: Perform feature point matching, matching point screening, and transformation matrix calculation to obtain the transformation matrix. Use the transformation matrix to perform affine transformation on the image to be aligned and align it to the reference image.

[0020] In the above scheme, the above step 1 specifically includes the following steps:

[0021] Step 1.1: Select an image with clear image, prominent device body and weak lighting changes as a reference image;

[0022] Step 1.2: Draw the location of the main equipment on the reference diagram by manually marking it with a rectangular frame.

[0023] In the above scheme, step 2 specifically includes the following steps:

[0024] Step 2.1: Select a target tracking algorithm and train it using data from the substation scenario to achieve better tracking results in the substation scenario;

[0025] Step 2.2: Input the reference image, the annotation box of the main device, and the image to be aligned as input to the target tracking algorithm, run the target tracking algorithm, and obtain the position box of the same device body in the image to be aligned.

[0026] In the above scheme, step 3 specifically includes the following steps:

[0027] Step 3.1: If the target tracking algorithm tracks multiple target boxes, it is necessary to cut out each device in the image to be aligned through the target box. The set of multiple devices is recorded as set A, A = {a1, a2, ..., an}. At the same time, the main device in the reference image is cut out according to the manually marked box, recorded as B.

[0028] Step 3.2: The similarity between each image in set A and B can be calculated by similarity calculation methods (such as texture similarity, structural similarity, twin network, etc.);

[0029] Step 3.3: If multiple similarity calculation methods are used, weighting or voting are used to select the image most similar to B, and the corresponding target box is used as the result box of target tracking.

[0030] In the above scheme, step 4 specifically includes the following steps:

[0031] Step 4.1: Perform feature point detection (such as SIFT, SURF, ORB, etc.) on the reference image and the image to be aligned.

[0032] Step 4.2: Use the manual annotation box to filter the key points of the reference image, and use the target tracking box to filter the key points in the image to be aligned. The key points in the target box are uniformly retained, and the key points outside the target box are discarded.

[0033] Step 4.2.1: Screen the key points of the reference image:

[0034] Obtain a manually marked box on the reference image, where the box marks the location of the subject device;

[0035] Traverse all feature points on the reference image and determine whether each feature point is within the manually marked box;

[0036] If the feature point is within the manually marked box, keep the feature point, otherwise discard the feature point;

[0037] After screening, a reference image key point set P1' located within the manually labeled frame is obtained.

[0038] Step 4.2.2: Screen the key points of the image to be aligned:

[0039] Obtain a target tracking frame on the image to be aligned, which is obtained through a target tracking algorithm and represents the position of the main device;

[0040] Traverse all feature points on the image to be aligned and determine whether each feature point is within the target tracking frame;

[0041] If the feature point is within the target tracking frame, keep the feature point, otherwise discard the feature point;

[0042] After screening, a set of key points P2' of the image to be aligned located in the target tracking frame is obtained.

[0043] In the above scheme, step 5 specifically includes the following steps:

[0044] Step 5.1, performing feature point matching on the reference image key point set P1' and the key point set P2' of the image to be aligned selected in step 4;

[0045] Step 5.2, filter the matching results obtained in step 5.1, and retain the feature point pairs with high matching quality;

[0046] Step 5.3, based on the feature point pairs selected in step 5.2, calculate a transformation matrix, which is used to transform the image to be aligned to a position aligned with the reference image;

[0047] Step 5.4: Use the transformation matrix calculated in step 5.3 to perform affine transformation on the image to be aligned, so as to align the image to be aligned with the reference image.

[0048] This proposal is suitable for solving the problem that the reference image is fixed and can be prepared in advance, and the image to be aligned is obtained in real time and cannot be manually intervened. In the early stage, it is only necessary to align the main device in the image without paying attention to the background. This proposal can effectively solve the shortcomings of existing solutions:

[0049] 1. It is also a local alignment in nature. The position of the device in the image to be aligned is determined by the target tracking algorithm. After obtaining the key points of the image, the key points can be filtered by the position information of the device. However, only two patches are actually used for alignment, which greatly reduces the amount of calculation.

[0050] 2. Compared with the optical flow method, the estimated pixel position changes using the target tracking method are more accurate.

[0051] 3. Compared with the global alignment method, there is no need to pay attention to the key points on the background. Only the key points on the device are selected to minimize the interference of background information and effectively align the position of the main device. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0053] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0054] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0055] The present invention innovatively applies the target tracking algorithm to the image alignment task, and quickly locates the position of the main device in the image to be aligned through target tracking, thereby converting the global image alignment problem into a local image alignment problem. The advantage of this is that the amount of calculation is reduced and the accuracy of alignment is improved, especially for the alignment of the main equipment in the substation scene.

[0056] The present invention provides an image alignment method based on target tracking in a substation scene, comprising the following steps:

[0057] Step 1: Use a rectangular frame to mark the location of the main equipment in the reference diagram by manual marking;

[0058] Step 2: Use the target tracking algorithm to obtain the position of the main device on the image to be aligned, that is, the frame obtained by target tracking;

[0059] Step 3: There may be other devices similar to the main device on the image to be aligned. In this case, it is necessary to compare the similarity of each tracked result with the main device on the reference image and retain the device position with the highest similarity;

[0060] Step 4: Detect feature points of the reference image and the image to be aligned, use the annotation box to filter the key points of the reference image, and use the target tracking box to filter the key points in the image to be aligned;

[0061] Step 5: Perform feature point matching, matching point screening, and transformation matrix calculation to obtain the transformation matrix. Use the transformation matrix to perform affine transformation on the image to be aligned and align it to the reference image.

[0062] In the above scheme, the above step 1 specifically includes the following steps:

[0063] Step 1.1: Select an image with clear image, prominent device body and weak lighting changes as a reference image;

[0064] Step 1.2: Draw the location of the main equipment on the reference diagram by manually marking it with a rectangular frame.

[0065] In the above scheme, step 2 specifically includes the following steps:

[0066] Step 2.1: Select a target tracking algorithm and train it using data from the substation scenario to achieve better tracking results in the substation scenario;

[0067] Step 2.2: Input the reference image, the annotation box of the main device, and the image to be aligned as input to the target tracking algorithm, run the target tracking algorithm, and obtain the position box of the same device body in the image to be aligned.

[0068] In the above scheme, step 3 specifically includes the following steps:

[0069] Step 3.1: If the target tracking algorithm tracks multiple target boxes, it is necessary to cut out each device in the image to be aligned through the target box. The set of multiple devices is recorded as set A, A = {a1, a2, ..., an}. At the same time, the main device in the reference image is cut out according to the manually marked box, recorded as B.

[0070] Step 3.2: The similarity between each image in set A and B can be calculated by similarity calculation methods (such as texture similarity, structural similarity, twin network, etc.);

[0071] Step 3.3: If multiple similarity calculation methods are used, weighting or voting are used to select the image most similar to B, and the corresponding target box is used as the result box of target tracking.

[0072] In the above scheme, step 4 specifically includes the following steps:

[0073] Step 4.1: Perform feature point detection (such as SIFT, SURF, ORB, etc.) on the reference image and the image to be aligned.

[0074] Step 4.2: Use the manual annotation box to filter the key points of the reference image, and use the target tracking box to filter the key points in the image to be aligned. The key points in the target box are uniformly retained, and the key points outside the target box are discarded.

[0075] Step 4.2.1: Screen the key points of the reference image:

[0076] Obtain a manually marked box on the reference image, where the box marks the location of the subject device;

[0077] Traverse all feature points on the reference image and determine whether each feature point is within the manually marked box;

[0078] If the feature point is within the manually marked box, keep the feature point, otherwise discard the feature point;

[0079] After screening, a reference image key point set P1' located within the manually labeled frame is obtained.

[0080] Step 4.2.2: Screen the key points of the image to be aligned:

[0081] Obtain a target tracking frame on the image to be aligned, which is obtained through a target tracking algorithm and represents the position of the main device;

[0082] Traverse all feature points on the image to be aligned and determine whether each feature point is within the target tracking frame;

[0083] If the feature point is within the target tracking frame, keep the feature point, otherwise discard the feature point;

[0084] After screening, a set of key points P2' of the image to be aligned located in the target tracking frame is obtained.

[0085] In the above scheme, step 5 specifically includes the following steps:

[0086] Step 5.1, performing feature point matching on the reference image key point set P1' and the key point set P2' of the image to be aligned selected in step 4;

[0087] Step 5.2, filter the matching results obtained in step 5.1, and retain the feature point pairs with high matching quality;

[0088] Step 5.3, based on the feature point pairs selected in step 5.2, calculate a transformation matrix, which is used to transform the image to be aligned to a position aligned with the reference image;

[0089] Step 5.4: Use the transformation matrix calculated in step 5.3 to perform affine transformation on the image to be aligned, so as to align the image to be aligned with the reference image.

[0090] The technical solution of this application proposal involves an image alignment method based on target tracking in a substation scene, which mainly includes the following key steps:

[0091] 1. Application of target tracking algorithm: The position of the main device on the image to be aligned is obtained through the target tracking algorithm. This step innovatively applies the target tracking algorithm to the image alignment task, quickly locating the position of the main device in the image to be aligned, thereby avoiding the huge amount of calculation in traditional local alignment.

[0092] 2. Similarity comparison: Perform similarity comparison between multiple target frames tracked on the image to be aligned and the main device on the reference image, and retain the device position with the highest similarity. This step helps ensure the accuracy of image alignment.

[0093] 3. Key point screening and matching: Feature point detection is performed on the reference image and the image to be aligned, and the key points are screened using the manual annotation box and the target tracking box, and the key points in the target box are uniformly retained. This step meets business needs by only using the key points on the main device for image alignment, while improving the overall business effect.

[0094] 4. Conventional steps of image alignment: including feature point matching, matching point screening, transformation matrix calculation, and using the transformation matrix to perform affine transformation on the image to be aligned and align it to the reference image. These steps are the conventional process of image alignment, which ensures the accuracy and efficiency of image alignment.

[0095] Compared with the existing technology, this application proposal has the following technical advantages:

[0096] Improved computational efficiency: The position of the device in the image to be aligned is determined by the target tracking algorithm. Only two patches are actually used for alignment, which greatly reduces the amount of computation.

[0097] Improved accuracy: Compared with the optical flow method, the pixel position changes estimated using the object tracking method are more accurate.

[0098] Reduced background interference: Compared with the global alignment method, there is no need to focus on the key points on the background. Only the key points on the device are selected to minimize the interference of background information and effectively align the position of the main device.

[0099] In summary, this application proposal improves the efficiency and accuracy of image alignment by transforming the global image alignment problem into a local image alignment problem and innovatively applying the target tracking algorithm. It is particularly suitable for the image alignment requirements of the main equipment in the substation scenario.

Claims

1. A target tracking-based image alignment method for a substation scene, characterized in that: The following steps are involved: Step 1: Use a rectangular frame to mark the location of the main equipment in the reference diagram; Step 2: Use the target tracking algorithm to obtain the position of the main device on the image to be aligned, that is, the frame obtained by target tracking; Step 3: Compare the similarity between each tracking result and the main device on the reference map, and retain the device position with the highest similarity; Step 4: Detect feature points of the reference image and the image to be aligned, use the annotation box to filter the key points of the reference image, and use the target tracking box to filter the key points in the image to be aligned; Step 5: Perform feature point matching, matching point screening, and transformation matrix calculation to obtain the transformation matrix. Use the transformation matrix to perform affine transformation on the image to be aligned and align it to the reference image.

2. The target tracking-based image alignment method for a substation scene according to claim 1 is characterized in that: The above step 1 specifically includes the following steps: Step 1.1: Select an image with clear image, prominent device body and weak lighting changes as a reference image; Step 1.2: Draw the location of the main equipment on the reference diagram by manually marking it with a rectangular frame.

3. The image alignment method based on target tracking in a substation scene according to claim 1 is characterized in that: Step 2 specifically includes the following steps: Step 2.1: Select a target tracking algorithm and train it using data from the substation scenario to achieve better tracking results in the substation scenario; Step 2.2: Input the reference image, the annotation box of the main device, and the image to be aligned as input to the target tracking algorithm, run the target tracking algorithm, and obtain the position box of the same device body in the image to be aligned.

4. The image alignment method based on target tracking in a substation scene according to claim 1 is characterized in that: Step 3 specifically includes the following steps: Step 3.1: If the target tracking algorithm tracks multiple target boxes, it is necessary to cut out each device in the image to be aligned through the target box. The set of multiple devices is recorded as set A, A = {a1, a2, ..., an}. At the same time, the main device in the reference image is cut out according to the manually marked box, recorded as B. Step 3.2: The similarity between each image in set A and B can be calculated by similarity calculation method; Step 3.3: If multiple similarity calculation methods are used, weighting or voting are used to select the image most similar to B, and the corresponding target box is used as the result box of target tracking.

5. The target tracking-based image alignment method for a substation scene according to claim 1 is characterized in that: Step 4 specifically includes the following steps: Step 4.1: Detect feature points on the reference image and the image to be aligned; Step 4.2: Use the manual annotation box to filter the key points of the reference image, and use the target tracking box to filter the key points in the image to be aligned. The key points in the target box are uniformly retained, and the key points outside the target box are discarded.

6. The method for image alignment based on target tracking in a substation scene according to claim 5 is characterized in that: Step 4.2 specifically includes the following steps: Step 4.2.1: Screen the key points of the reference image: Obtain a manually marked box on the reference image, where the box marks the location of the subject device; Traverse all feature points on the reference image and determine whether each feature point is within the manually marked box; If the feature point is within the manually marked box, keep the feature point, otherwise discard the feature point; After screening, a reference image key point set P1' located within the manually labeled frame is obtained. Step 4.2.2: Screen the key points of the image to be aligned: Obtain a target tracking frame on the image to be aligned, which is obtained through a target tracking algorithm and represents the position of the main device; Traverse all feature points on the image to be aligned and determine whether each feature point is within the target tracking frame; If the feature point is within the target tracking frame, keep the feature point, otherwise discard the feature point; After screening, a set of key points P2' of the image to be aligned located in the target tracking frame is obtained.

7. According to the target tracking-based image alignment method for a substation scene according to claim 6, step 5 specifically comprises the following steps: Step 5.1, performing feature point matching on the reference image key point set P1' and the key point set P2' of the image to be aligned selected in step 4; Step 5.2, filter the matching results obtained in step 5.1, and retain the feature point pairs with high matching quality; Step 5.3, based on the feature point pairs selected in step 5.2, calculate a transformation matrix, which is used to transform the image to be aligned to a position aligned with the reference image; Step 5.4: Use the transformation matrix calculated in step 5.3 to perform affine transformation on the image to be aligned, so as to align the image to be aligned with the reference image.

Images