Satellite-unmanned aerial vehicle integrated image processing and matching method

Abstract

The invention is suitable for the technical field of image matching, and provides a satellite-unmanned aerial vehicle integrated image processing and matching method based on an image matching and re-recognition system. The system comprises an image input module, an image preprocessing module, a feature extraction and matching module, a multi-thread processing module, a result output and visualization module and a system control and management module. The method comprises the following steps: inputting an image; preprocessing the image; feature extraction and matching; performing multi-thread processing; outputting and visualizing a result; and controlling and managing. The method is wide in application; the intelligent level of the monitoring system is improved in the scenes of safety monitoring, border patrol and the like; in traffic monitoring and management, the efficiency and safety of traffic management are improved; real-time monitoring and early warning of environment changes and disaster dynamic states are realized, and the emergency response capability is improved. In a pilotless automobile and an automatic system, the autonomous decision-making level of the system is improved; in military reconnaissance and intelligence analysis, the accuracy of combat command is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image matching, and particularly relates to a satellite - UAV integrated image processing and matching method. Background Art

[0002] In modern image matching applications, cross - domain image matching technology is crucial, especially in fields such as satellite imagery and UAV video surveillance. Existing image matching methods such as SIFT (Scale - Invariant Feature Transform), SURF (Speeded - Up Robust Features), etc., mainly rely on feature point extraction and matching algorithms. These methods achieve feature matching between different images by detecting key points in the images and extracting their descriptors. However, with the complication of application scenarios, especially in cross - domain environments (such as satellite images and UAV video images), there are significant visual differences, including resolution, viewing angle, lighting conditions, etc. These differences pose great challenges for existing image matching methods in identifying and matching the same or related image content. When dealing with small targets and moving targets, existing image matching methods also perform poorly. Since small targets occupy a limited number of pixels in the image and usually lack sufficient detail and texture information, it becomes more difficult to extract effective features. In a complex background, small targets are easily occluded by other objects and are vulnerable to background noise interference, further reducing the accuracy of matching. For moving targets, since their positions change at different times and viewing angles, it becomes infeasible to accurately locate based on the positional relationship of surrounding ground objects. In addition, existing image matching methods mostly adopt a single - thread processing mode. This mode has problems of slow processing speed and insufficient resource utilization in applications that process a large number of images or require real - time response. The single - thread mode cannot fully utilize the computing power of modern multi - core CPUs, resulting in low processing efficiency in high - load environments and being difficult to meet the requirements of real - time monitoring and intelligent detection. At the same time, the single - thread mode also has limitations in task management and exception handling, making it difficult to handle complex matching tasks and potential processing exceptions, affecting the stability and reliability of the system.

[0003] Current image matching products on the market usually consist of core modules such as image preprocessing, feature extraction and description, feature matching, and result processing and output. The image preprocessing module performs basic processing on the input satellite images and drone images to reduce the visual differences between different images. The feature extraction and description module uses algorithms such as SIFT and SURF to extract key points and their descriptors. The feature matching module then performs the matching and alignment of feature points to generate a set of matching points. Finally, the result processing and output module performs subsequent processing on the matching results to achieve accurate positioning and monitoring of the target. However, due to the limitations of feature extraction and the single-threaded implementation of the matching algorithm, existing products are difficult to achieve efficient and accurate matching and re-identification when dealing with small targets and moving targets. At the same time, in response to the requirements of real-time monitoring, the processing speed of existing products cannot meet the high-frequency image updates, resulting in matching delays and untimely responses, which limits their application effects in dynamic monitoring environments.

[0004] In summary, the existing technologies have deficiencies in aspects such as image matching speed, resource utilization rate, accuracy of small target detection, satellite and drone image preprocessing, and visualization of selected areas after matching. There is an urgent need for new solutions to break through these bottlenecks. For this reason, the present invention proposes a satellite-drone integrated image processing and matching method. Summary of the Invention

[0005] The purpose of the present invention is to provide a satellite-drone integrated image processing and matching method, aiming to solve the problems raised in the above background technology.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A satellite-drone integrated image processing and matching method, the method is based on an image matching and re-identification system, and the image matching and re-identification system includes an image input module, an image preprocessing module, a feature extraction and matching module, a multi-threaded processing module, a result output and visualization module, and a system control and management module; the method includes the following steps:

[0008] Step 1: Image input;

[0009] Load two types of image data, satellite images and drone images, through the image input module;

[0010] Step 2: Image preprocessing;

[0011] The image enters the image preprocessing module. In the image preprocessing module, the satellite image first undergoes normalization processing to standardize the pixel value range; subsequently, a quadrilateral region of interest is selected from the normalized satellite image, and the region is rotated, the contrast is enhanced, and the brightness is optimized to generate a preprocessed image and a corresponding transformation matrix;

[0012] Step 3: Feature Extraction and Matching;

[0013] The processed image enters the feature extraction and matching module. In the feature extraction and matching module, key feature points in the satellite image and the UAV image are extracted, and feature matching is performed to generate a set of matching points;

[0014] Step 4: Multithreaded Processing;

[0015] The parallel processing of the image matching task is carried out through the multithreaded processing module;

[0016] Step 5: Result Output and Visualization;

[0017] In the result output and visualization module, the matched polygon area is mapped onto the UAV image, and an image processing tool is used to draw the visualized matching result; the generated visualized image is saved to the specified path, and at the same time, the coordinate information and the time-consuming data of each step during the matching process are recorded and printed;

[0018] Step 6: Control and Management;

[0019] The system control and management module is used to coordinate the work of each module in the main control process, manage the loading, preprocessing, matching, and result output of the images, and record the running time of the entire program.

[0020] Furthermore, in the above Step 1, the satellite image is read from the specified path and loaded into the memory using the cv2.imread function of OpenCV as the reference image for matching; at the same time, the UAV images are read one by one from the predefined image list for batch processing.

[0021] Furthermore, the specific process of the above Step 2 is as follows:

[0022] In the image preprocessing module, first, the satellite image is normalized, and the pixel values of the image are standardized to the range of 0 - 255 using the cv2.normalize function; subsequently, the select_quad function is called to select the quadrilateral region of interest quad_ori from the normalized satellite image; then, the preprocessing function is used to perform rotation adjustment, contrast enhancement, and brightness optimization on the image to generate the preprocessed image img_prepro and the corresponding transformation matrix matrix; then, the rotate_quad function is used to rotate the original quadrilateral coordinates according to the transformation matrix to obtain the preprocessed quadrilateral coordinates quad_prepro.

[0023] Furthermore, the process of selecting the quadrilateral region is as follows:

[0024] Create a copy of the image:

[0025] Equation 1: I′ = I;

[0026] Where: I′ is the transformed image matrix; I is the original image matrix: I ∈ R H×W×C , where H is the height, W is the width, and C is the number of channels;

[0027] Click on four points in the satellite image window in sequence and record their coordinates:

[0028] Equation 2: P = {P 1 , P 2 , P 3 , P 4};

[0029] Where: P is the set of vertex coordinates;

[0030] For each newly selected point P i , draw the previous dot (x i , y i ) on I′. For each pair of consecutive points P i-1 and P i , draw a line segment (P i-1 , P i ) on I′. When all four points are selected, draw the last line segment connecting P 4 and P 1 , draw the line segment (P 4 , P 1 ). The quadrilateral is represented as P 1 →P 2 →P 3 →P 4 →P 1 . The drawn image copy is I′ and the vertex coordinate array P, output (I′, P).

[0031] Furthermore, the specific process of the step of using the preprocessing function to perform rotation adjustment, contrast enhancement, and brightness optimization on the image to generate the preprocessed image img_prepro and the corresponding transformation matrix matrix is as follows:

[0032] Use the cv2.convertScaleAbs function to adjust the contrast and brightness of the image according to the given contrast and brightness parameters; generate a perspective transformation matrix by calling the get_matrix function to convert the input image to the specified size and perspective. The formula for the obtained perspective transformation matrix is as follows:

[0033] Equation 3: T = T t × T s × Tyaw ;

[0034] In the formula: T is a 3x3 matrix, which is a comprehensive transformation matrix and includes three parts: rotation, scaling, and translation; T t is the translation matrix; T s is the scaling matrix; T yaw is the rotation matrix;

[0035] Among them, the rotation matrix T yaw is used to rotate the image around the origin by a specified angle:

[0036] Formula 4:

[0037] In the formula: is the radian value of the rotation angle;

[0038] The scaling matrix T s is as follows:

[0039] Formula 5:

[0040] In the formula: s is the scaling factor, and the calculation method is:

[0041] Formula 6:

[0042] In the formula: x max , x min are respectively the maximum and minimum values of the abscissas of the four selected points; y max , y min are respectively the maximum and minimum values of the ordinates of the four selected points; size is the size of the output image;

[0043] The translation matrix T t is as follows:

[0044] Formula 7:

[0045] In the formula: t x is the translation amount of the image in the x direction; t y is the translation amount of the image in the y direction; the calculation method is as follows:

[0046] Formula 8:

[0047] According to Formula 3, after expansion, we get:

[0048] Formula 9:

[0049] Use the warpPerspective function in OpenCV to perform perspective transformation on the adjusted image I′ through the transformation matrix T to obtain the output image I output .

[0050] Further, the specific process of the step of using the rotate_quad function to rotate the original quadrilateral coordinates according to the transformation matrix to obtain the preprocessed quadrilateral coordinates quad_prepro is as follows:

[0051] The formula is:

[0052] Equation 10:

[0053] In the formula: x i ′ is the abscissa of the vertex of the output quadrilateral; y i ′ is the ordinate of the vertex of the output quadrilateral; w i ′ is the scale factor; T is the transformation matrix, and the vertices of the input quadrilateral are P = {(x 1 , y 1 ), (x 2 , y 2 ), (x 3 , y 3 ), (x 4 , y 4 )}, then the vertices of the output quadrilateral are P′ = {(x 1 ′, y 1 ′), (x 2 ′, y 2 ′), (x 3 ′, y 3 ′), (x 4 ′, y 4 ′)}, and after normalization, it is obtained:

[0054] Equation 11:

[0055] Further, the specific process of the step 3 is as follows:

[0056] In the feature extraction and matching module, first instantiate an object xfeat of the XFeat class to perform efficient feature extraction and matching operations; load the preprocessed satellite image data image0 and transfer it to the CUDA device; then, traverse the list of drone images, and perform the following process on each drone image: call the match_with_prepro function, first load and resize the drone image, and then transfer it to the CUDA device; through the xfeat.match_xfeat method, extract and match feature points from the satellite image and the drone image to obtain the matching point sets points0 and points1; if the number of matching points is insufficient, skip the current image;

[0057] After successfully matching the feature points, the transform_polygon_global function is called. First, the cv2.findHomography function in OpenCV is used to calculate the homography matrix from the source point set pts0 to the target point set pts1 through the MAGSAC algorithm; if the matrix calculation fails, that is, matrix is None, the function will return (0,0), indicating that the transformation is unsuccessful. Otherwise, the function uses the cv2.perspectiveTransform function to perform a perspective transformation on the input polygon vertices according to the calculated homography matrix, generating new polygon vertex coordinates, and returns in the form of (1, transformed_polygon), indicating that the transformation is successful and its result; the transformed polygon quad_img1 is generated;

[0058] Next, the convert_position function is used to convert the polygon position according to the scaling ratio, and the shrink_polygon function is used to perform a shrinking process on the polygon to obtain shrink_quad_1_ori; the cv2.boundingRect function in OpenCV is used to calculate the coordinates (x, y, w, h) of the minimum bounding rectangle, and the polygon coordinates are rounded and converted into a list of integers.

[0059] Further, the specific process of step 4 is as follows:

[0060] The multi-threaded processing module realizes the parallel processing of image matching tasks through the ThreadPool.py function. The ThreadPool.py function traverses the input image list and the corresponding paths, and simultaneously processes the matching tasks of multiple UAV images through the thread pool.

[0061] Further, the specific process of step 5 is as follows:

[0062] In the result output and visualization module, the cv2.polylines function in OpenCV is used to draw the transformed polygon area on the UAV image; the generated visualization result image is saved to the specified path through the cv2.imwrite function; at the same time, the system prints out the converted coordinate information (x, y, w, h) and the time-consuming information of each step.

[0063] Compared with the prior art, the beneficial effects of the present invention are:

[0064] The method proposed by the present invention demonstrates broad application potential. In scenarios such as security monitoring and border patrol, this method can achieve efficient and accurate target detection and recognition, enhancing the intelligence level of the monitoring system; in traffic monitoring and management, this method can identify and track dynamic targets such as vehicles and pedestrians in real time, effectively improving the efficiency and safety of traffic management; by combining satellite and drone images, this method can achieve real-time monitoring and early warning of environmental changes and disaster dynamics, enhancing the emergency response ability; in driverless cars and automation systems, this method provides high-precision environmental perception and target recognition, enhancing the autonomous decision-making level of the system; in military reconnaissance and intelligence analysis, by combining satellite and drone images, this method can achieve efficient target recognition and situation awareness, improving the accuracy of combat command. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flowchart of the method of the present invention.

[0066] Figure 2 It is the matching effect diagram in Embodiment 1.

[0067] Figure 3 It is the effect diagram of obtaining the original coordinate points in Embodiment 1.

[0068] Figure 4 It is the coordinate information after conversion, image loading time, matching time, output time and total time information in Embodiment 1.

[0069] Figure 5 It is the schematic diagram of the output image file name in Embodiment 1.

[0070] Figure 6 It is the matching result effect diagram in Embodiment 1.

[0071] Figure 7 It is the thread pool parallel processing in Embodiment 1.

[0072] Figure 8 It is the effect diagram of obtaining the original coordinate points in Embodiment 2.

[0073] Figure 9 It is the matching result effect diagram in Embodiment 2. DETAILED DESCRIPTION OF THE INVENTION

[0074] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present invention, the technical solution of the present invention will be described in detail below, but it should not be construed as a limitation on the scope of implementation of the present invention.

[0075] The following describes the specific implementation of the present invention in detail with reference to specific embodiments.

[0076] An embodiment of the present invention provides a satellite - UAV integrated image processing and matching method, which is based on an image matching and re - identification system. The image matching and re - identification system includes an image input module, an image pre - processing module, a feature extraction and matching module, a multi - thread processing module, a result output and visualization module, and a system control and management module. This solution has two application modes: one is to perform offline processing on the frames of images transmitted back by the UAV at the device end; the other is to be installed on the UAV's on - board computer and perform real - time monitoring frame by frame through a camera. And two modes, namely single - thread in base.py and multi - thread in ThreadPool.py, are set, which can be appropriately selected according to resources.

[0077] The design idea of this solution is as follows:

[0078] In the field of image matching, in the face of the challenges of detection and matching of small targets in cross - domain environments due to significant visual differences, less pixel information, and being easily occluded, etc., the present invention has developed an image feature extraction and matching algorithm based on XFeat. By loading and pre - processing satellite and UAV images, the match_xfeat function of XFeat is used for efficient feature matching, and multi - scale feature extraction and attention mechanism are introduced, significantly enhancing the recognition and matching accuracy of small targets. Aiming at the re - identification challenges brought by the position changes of moving targets at different times and perspectives, the present invention designs a re - identification process based on feature point matching and geometric transformation. Using global transformation and coordinate conversion, accurate re - identification of moving targets in cross - domain images is achieved, reducing the impact of pose changes and occlusions on the recognition results. In addition, considering the requirements of limited hardware resources in actual engineering applications, a lightweight matching and re - identification algorithm is constructed. By using CUDA acceleration and optimized algorithm processes, a balance between algorithm performance and computational efficiency is achieved, ensuring real - time application capabilities on end - side devices. At the same time, two processing modes, single - thread and multi - thread, are provided, further improving the processing speed and resource utilization rate of the system, enhancing the responsiveness and stability of the program, and meeting the needs of real - time monitoring and intelligent detection.

[0079] The flow chart of this method is as Figure 1 shown, and specifically includes the following steps:

[0080] Step 1: Load two types of image data through the image input module: satellite images and UAV images. The satellite images are read through the specified path and loaded into memory using the cv2.imread function of OpenCV as the reference images for matching. At the same time, the UAV images are read one by one from the predefined image list (such as ['01', '02',...,'50']) to obtain the corresponding image file paths for batch processing.

[0081] Step 2: The image enters the image preprocessing module. In the image preprocessing module, first, the satellite image is normalized. The cv2.normalize function is used to standardize the image pixel values to the range of 0 - 255 to improve the image quality and reduce the influence of illumination and contrast differences.

[0082] Subsequently, the quadrilateral region of interest quad_ori is selected from the normalized satellite image by calling the select_quad function. The process of selecting the quadrilateral region is as follows:

[0083] Create a copy of the image:

[0084] Equation 1: I′ = I;

[0085] Where: I′ is the transformed image matrix; I is the original image matrix: I ∈ R H×W×C , where H is the height, W is the width, and C is the number of channels (usually 3, representing RGB).

[0086] Click four points in the satellite image window in sequence and record their coordinates:

[0087] Equation 2: P = {P 1 , P 2 , P 3 , P 4};

[0088] Where: P is the vertex coordinate group;

[0089] For each newly selected point P i , draw the previous dot (x i , y i ) on I′. For each pair of consecutive points P i-1 and P i , draw a line segment (P i-1 , P i ) on I′. When four points are all selected, draw the last line segment connecting P 4 and P 1 , draw the line segment (P 4 , P 1 ). The quadrilateral can be represented as P 1 → P 2 → P 3 → P 4 → P 1 . The drawn image copy is I′ and the vertex coordinate array P, and the output is (I′, P).

[0090] Then, use the preprocessing function to rotate and adjust the image, enhance the contrast (contrast = 1.3), and optimize the brightness (brightness = 25) to generate the preprocessed image img_prepro and the corresponding transformation matrix matrix. The specific method is to use the existing cv2.convertScaleAbs function to adjust the contrast and brightness of the image according to the given contrast and brightness parameters. Generate a perspective transformation matrix (matrix) by calling the get_matrix function to convert the input image to the specified size (size) and viewing angle (yaw). The formula for the obtained perspective transformation matrix is as follows:

[0091] Equation 3: T = T t ×T s ×T yaw ;

[0092] Where: T is a 3x3 matrix, the comprehensive transformation matrix, which includes three parts: rotation, scaling, and translation; T t is the translation matrix; T s is the scaling matrix; T yaw is the rotation matrix;

[0093] Among them, the rotation matrix T yaw is used to rotate the image around the origin by a specified angle.

[0094] Equation 4:

[0095] Where: is the radian value of the rotation angle;

[0096] The scaling matrix T s is as follows:

[0097] Equation 5:

[0098] Where: s is the scaling factor, and the calculation method is:

[0099] Equation 6:

[0100] Where: x max , x min are the maximum and minimum values of the abscissas of the four selected points respectively; y max , y min are the maximum and minimum values of the ordinates of the four selected points respectively; size is the size of the output image (can be customized).

[0101] The translation matrix T t is as follows:

[0102] Equation 7:

[0103] Where: t x is the translation amount of the image in the x direction; t y is the translation amount of the image in the y direction; The calculation method is as follows:

[0104] Equation 8:

[0105] According to Equation 3, after expansion, we get:

[0106] Equation 9:

[0107] Use the warpPerspective function of OpenCV to perform perspective transformation on the adjusted image I′ through the transformation matrix T to obtain the output image I output .

[0108] Next, use the rotate_quad function to rotate the original quadrilateral coordinates according to the transformation matrix to obtain the preprocessed quadrilateral coordinates quad_prepro, providing an accurate reference area for subsequent feature matching. The formula is:

[0109] Equation 10:

[0110] Where: x i ′ is the abscissa of the vertex of the output quadrilateral; y i ′ is the ordinate of the vertex of the output quadrilateral; w i ′ is the scale factor; T is the transformation matrix, and the vertices of the input quadrilateral are P = {(x 1 , y 1 ), (x 2 , y 2 ), (x 3 , y 3 ), (x 4 , y 4 )}, then the vertices of the output quadrilateral are P′ = {(x 1 ′, y 1 ′), (x 2 ′, y 2 ′), (x 3 ′, y 3 ′), (x 4 ′, y 4 ′)}, and after normalization, we get:

[0111] Equation 11:

[0112] Through these preprocessing steps, the consistency and high quality of the selected region among different domains are ensured, providing an accurate reference for subsequent feature matching.

[0113] Step 3: The processed image enters the feature extraction and matching module. In the feature extraction and matching module, first, an object xfeat of the XFeat class is instantiated to perform efficient feature extraction and matching operations. The preprocessed satellite image data image0 is loaded and transferred to the CUDA device to accelerate the subsequent feature matching process. Subsequently, the system traverses the list of drone images and performs the following process for each drone image: The match_with_prepro function is called. First, the drone image is loaded and resized, and then it is transferred to the CUDA device. Through the xfeat.match_xfeat method, the system extracts and matches feature points from the satellite image and the drone image, obtaining the matching point sets points0 and points1. Through an efficient feature matching algorithm, the system can accurately identify and match the same or related content in cross-domain images. If the number of matching points is insufficient, the system will skip the current image to ensure the reliability of the matching result.

[0114] After successfully matching the feature points, the system calls the transform_polygon_global function. Specifically, the function first uses the cv2.findHomography function in OpenCV to calculate the homography matrix from the source point set pts0 to the target point set pts1 through the MAGSAC algorithm (a robust estimation algorithm). If the matrix calculation fails (i.e., matrix is None), the function will return (0,0), indicating that the transformation is unsuccessful. Otherwise, the function will use the cv2.perspectiveTransform function to perform a perspective transformation on the input polygon vertices according to the calculated homography matrix, generating new polygon vertex coordinates and returning them in the form of (1,transformed_polygon), indicating the success of the transformation and its result. The transformed polygon quad_img1 is generated.

[0115] Next, the convert_position function is used to convert the polygon position according to the scaling ratio, and the shrink_polygon function is used to shrink the polygon by a certain ratio to obtain shrink_quad_1_ori. The coordinates (x,y,w,h) of the minimum bounding rectangle are calculated using the cv2.boundingRect function in OpenCV, and the polygon coordinates are rounded and converted into a list of integers for subsequent visualization processing.

[0116] Step 4: To improve the processing efficiency, the multi-thread processing module adopts a parallel processing strategy and implements the parallel processing of image matching tasks through the ThreadPool.py function. This module uses batch processing and multi-thread technologies to efficiently process a set of images. Specifically, the ThreadPool.py function traverses the input image list and corresponding paths, and simultaneously processes the matching tasks of multiple UAV images through the thread pool, making full use of multi-core CPU and GPU resources, significantly improving the processing speed and resource utilization rate. This strategy not only enables the system to process multiple UAV images simultaneously, greatly shortening the overall processing time and improving the system's processing speed, but also optimizes the system's resource utilization rate, meeting the high-frequency and high-throughput image processing requirements in real-time monitoring and intelligent detection applications.

[0117] Step 5: In the result output and visualization module, the cv2.polylines function of OpenCV is used to draw the transformed polygon area (blue lines) on the UAV image to visually display the matching effect. The generated visualization result image is saved to the specified path through the cv2.imwrite function (such as. / university_results / MatchPre_out_time string.png) for users to verify and analyze the matching accuracy. In addition, the system also prints out the transformed coordinate information (x, y, w, h) and the time consumption information of each step to help optimize the system performance and further improve the matching algorithm.

[0118] Step 6: The system control and management module coordinates the working processes of each module in the main function, manages the loading, preprocessing, matching, and result output of images, ensuring the smooth operation of the entire system process. This module is also responsible for recording the running time of the entire program, especially the performance improvement in multi-thread mode, ensuring that the system can still operate stably and efficiently in a high-load environment.

[0119] Example 1: To test the running effect of the present invention, the satellite-UAV dataset university-1652 of the University of Technology Sydney is used for testing. According to the above steps, the processed satellite image is matched with one UAV image. At this time, the selected area operation is not performed first, and the preprocessing effect of the satellite image before matching and the feasibility of matching are proved first. The specific effect is as Figure 2 , it can be seen that the image preprocessing effect and matching feasibility of the present invention both meet the expectations.

[0120] Next, the single-thread mode is tested. First, the UAV images of university-1652 are organized into a folder to prepare for reading the UAV images one by one. Then, an area is selected on the satellite image, and 4 coordinates are returned. The effect is as Figure 3 shown.

[0121] Then, adjust the brightness and contrast, perform perspective transformation, and match each resized and formatted drone image with the satellite image one by one. In the present invention, record the time of each processing step, including the picture loading event, matching time, output time, and the total time for each match. By recording the time consumption of each key step (such as image loading, preprocessing, feature matching, coordinate transformation, and result output), the system can monitor the running efficiency of each module in real time, which helps to promptly detect performance bottlenecks, ensure that the system can still operate efficiently in a high-load environment, and optimize and improve a certain step. The printed information of the transformed coordinate information, picture loading time, matching time, output time, and total time is as Figure 4 shown. By recording and optimizing the time consumption of each processing step, the system can provide a faster response speed and the efficiency of the overall process.

[0122] After all drone images are matched, view the output results at the previously set path. The file name is the output moment time (year, month, day, hour, minute, second), which is convenient for subsequent recording, as Figure 5 shown in the schematic diagram of the output picture file name.

[0123] View the matching results, as Figure 6 shown. It can be seen that after matching, in the case of the drone at different angles, different heights, and different distances, the quadrilateral area drawn on the satellite image can be well mapped to the drone image, and the transformed coordinates can be obtained. The drone can perform subsequent operations according to the task requirements.

[0124] Next, adopt the multi-threaded mode, introduce a thread pool, and load the drone images and perform matching simultaneously. The effect is as Figure 7 shown. The matching time is longer than that in the single-threaded mode because all pictures in the thread pool are being matched, and the time calculated is the matching time of all pictures.

[0125] Embodiment 2: Given that the matching effect of offline processing is better, we configure the project on the on-board computer of the drone, equipped with a high-definition camera. Before taking off, select an area on the satellite image and obtain the coordinates, and then perform preprocessing, as Figure 8 shown. Subsequently, take off the drone, and match each frame of the image read by the drone camera with the satellite image according to the selected area on the satellite image and the preprocessing result. The matching results are saved in the background. After the operation ends, the drone returns, and the matching results can be viewed in the background, as Figure 9 shown. It can be seen that after processing and matching, the quadrilateral area in the satellite image is accurately matched to various angle pictures taken by the drone, and the matching accuracy is considerable.

[0126] The above are only the preferred embodiments of the present invention. It should be noted that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicability of the patent.

Claims

1. A satellite-UAV integrated image processing and matching method, characterized in that: The method is based on an image matching and re-identification system, which includes an image input module, an image preprocessing module, a feature extraction and matching module, a multi-thread processing module, a result output and visualization module, and a system control and management module; the method includes the following steps: Step 1: Image input; Load two types of image data, satellite images and drone images, through the image input module; Step 2: Image preprocessing; The image enters the image preprocessing module, in which the satellite image is first normalized to standardize the pixel value range; then, the quadrilateral area of ​​interest is selected from the normalized satellite image, and the area is rotated, contrast enhanced, and brightness optimized to generate the preprocessed image and the corresponding transformation matrix; Step 3: Feature extraction and matching; The processed images enter the feature extraction and matching module, where key feature points in satellite images and drone images are extracted, and feature matching is performed to generate a matching point set; Step 4: Multithreading; Parallel processing of image matching tasks through multi-threaded processing modules; Step 5: Result output and visualization; In the result output and visualization module, the matched polygonal area is mapped to the drone image, and the image processing tool is used to draw the visual matching results; the generated visual image is saved to the specified path, and the coordinate information of the matching process and the time-consuming data of each step are recorded and printed; Step 6: Control and management; The system control and management module is used to coordinate the work of each module in the main control process, manage the loading, preprocessing, matching and result output of images, and record the running time of the entire program.

2. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: In step 1, the satellite image is read from the specified path and loaded into the memory using the cv2.imread function of OpenCV as the matching reference image; at the same time, the drone images are read one by one from the predefined image list in preparation for batch processing.

3. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The specific process of step 2 is as follows: In the image preprocessing module, the satellite image is first normalized, and the image pixel values ​​are standardized to the range of 0-255 using the cv2.normalize function; then, the quadrilateral region of interest quad_ori is selected from the normalized satellite image by calling the select_quad function; then, the image is rotated, contrast enhanced, and brightness optimized using the preprocessing function to generate the preprocessed image img_prepro and the corresponding transformation matrix matrix; then, the rotate_quad function is used to rotate the original quadrilateral coordinates according to the transformation matrix to obtain the preprocessed quadrilateral coordinates quad_prepro.

4. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The process of selecting the quadrilateral region is as follows: Create a copy of the image: Formula 1: I′=I; Where: I′ is the transformed image matrix; I is the original image matrix: I∈R H×W×C , where H is the height, W is the width, and C is the number of channels; Click four points in the satellite image window in turn and record their coordinates: Formula 2: P = {P1, P2, P3, P4}; Where: P is the vertex coordinate group; For each newly selected point P i , draw a dot (x i ,y i ), for every pair of consecutive points P i-1 and P i , draw a line segment (P i-1 ,P i ), when all four points are selected, draw the last line segment connecting P4 and P1, draw the line segment (P4, P1), the quadrilateral is represented as P1→P2→P3→P4→P1, the copy of the image after drawing is I′ and the vertex coordinate array P, and output (I′, P).

5. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The specific process of using the preprocessing function to rotate, enhance contrast and optimize brightness of the image to generate the preprocessed image img_prepro and the corresponding transformation matrix matrix is ​​as follows: Use the cv2.convertScaleAbs function to adjust the contrast and brightness of the image according to the given contrast and brightness parameters; generate a perspective transformation matrix by calling the get_matrix function to convert the input image to the specified size and viewing angle. The formula of the perspective transformation matrix is ​​as follows: Formula 3: T = T t ×T s ×T yaw ; Where: T is a 3x3 matrix, which is a comprehensive transformation matrix, including rotation, scaling and translation; T t is the translation matrix; T s is the scaling matrix; T yaw is the rotation matrix; The rotation matrix T yaw Used to rotate an image by a specified angle around the origin: Formula 4: Where: is the radian value of the rotation angle; Scaling matrix T s as follows: Formula 5: Where: s is the scaling factor, calculated as: Formula 6: Where: x max ,x min are the maximum and minimum values ​​of the horizontal coordinates of the four selected points; max ,y min are the maximum and minimum values ​​of the ordinates of the four selected points respectively; size is the size of the output image; Translation matrix T t as follows: Formula 7: Where: t x is the translation of the image in the x direction; t y is the translation of the image in the y direction; the calculation method is as follows: Formula 8: According to formula 3, after expansion, we get: Formula 9: Use OpenCV's warpPerspective function to transform the adjusted image I' through the transformation matrix T to obtain the output image I output .

6. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The specific process of using the rotate_quad function to rotate the original quadrilateral coordinates according to the transformation matrix to obtain the pre-processed quadrilateral coordinates quad_prepro step is as follows: The formula is: Formula 10: Where: x i ′ is the horizontal coordinate of the output quadrilateral vertex; y i ′ is the vertical coordinate of the output quadrilateral vertex; w i ′ is the scale factor; T is the transformation matrix. The vertices of the input quadrilateral are P = {(x1, y1), (x2, y2), (x3, y3), (x4, y4)}, and the vertices of the output quadrilateral are P' = {(x1', y1'), (x2', y2'), (x3', y3'), (x4', y4')}. After normalization, we get: Formula 11:

7. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The specific process of step 3 is as follows: In the feature extraction and matching module, the object xfeat of the XFeat class is first instantiated to perform efficient feature extraction and matching operations; the preprocessed satellite image data image0 is loaded and transferred to the CUDA device; then, the drone image list is traversed, and the following process is performed for each drone image: the match_with_prepro function is called to first load and resize the drone image, and then transfer it to the CUDA device; through the xfeat.match_xfeat method, feature points are extracted and matched from the satellite image and the drone image to obtain the matching point sets points0 and points1; if the number of matching points is insufficient, the current image is skipped; After successfully matching the feature points, call the transform_polygon_global function. First, use OpenCV's cv2.findHomography function to calculate the homography matrix from the source point set pts0 to the target point set pts1 through the MAGSAC algorithm. If the matrix calculation fails, that is, matrix is ​​None, the function will return (0,0), indicating that the transformation is unsuccessful. Otherwise, the function uses the cv2.perspectiveTransform function to perform perspective transformation on the input polygon vertices according to the calculated homography matrix, generate new polygon vertex coordinates, and return them in the form of (1, transformed_polygon), indicating that the transformation is successful and its results. Generate the transformed polygon quad_img1. Next, convert_position function is used to convert the polygon position according to the scaling ratio, and shrink_polygon function is used to shrink the polygon to obtain shrink_quad_1_ori. OpenCV's cv2.boundingRect function is used to calculate the coordinates (x, y, w, h) of the minimum enclosing rectangle, and the polygon coordinates are rounded off and converted into an integer list.

8. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The specific process of step 4 is as follows: The multi-threaded processing module uses the ThreadPool.py function to realize parallel processing of image matching tasks. The ThreadPool.py function traverses the input image list and the corresponding path, and processes the matching tasks of multiple drone images simultaneously through the thread pool.

9. The satellite-UAV integrated image processing and matching method according to claim 1, characterized in that: The specific process of step 5 is as follows: In the result output and visualization module, the cv2.polylines function of OpenCV is used to draw the transformed polygonal area on the drone image; the generated visualization result image is saved to the specified path through the cv2.imwrite function; at the same time, the system prints out the converted coordinate information (x, y, w, h) and the time-consuming information of each step.

Images