A real-time stitching method and system based on drone-based forensic images

Abstract

The present invention discloses a method and system for real-time stitching of forensic images collected by drones. The method includes: a drone equipped with two depth cameras and a GPS signal collector; 3 minutes after the drone performs a task, a depth camera health detection algorithm determines whether the depth cameras are working properly. If they are not working properly, the information is reported to the command center; if the depth cameras are working properly, the main image and auxiliary image at the current moment are obtained, the boundaries of the overlapping areas are predicted, and the boundaries of the overlapping areas are verified; if the predicted overlapping area boundaries are inconsistent with the verified overlapping area boundaries, the random step algorithm is used to correct the overlapping area boundaries to obtain an accurate overlapping area, thereby completing the task of image stitching. The present invention first determines the performance of the depth camera through the depth camera health detection algorithm, and then finds the accurate overlapping area through the prediction-verification-correction method to complete the task of real-time stitching of forensic images collected by drones.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching, and in particular to a real-time stitching method and system for unmanned aerial vehicle (UAV)-based forensic imagery. Background Art

[0002] With the development of the economy and society, the need for evidence collection is gradually increasing. At present, cases in most parts of my country are still collected manually, such as water pollution, illegal construction and other issues. The evidence collection process is not only very time-consuming, but also has problems such as long evidence collection cycle, high labor costs and certain life safety risks faced by evidence collectors. Therefore, a new way to complete the evidence collection task is urgently needed.

[0003] After continuous development, drone technology has achieved many results in patrolling, monitoring and other aspects. However, the existing solutions still have many shortcomings. On the one hand, drone image stitching technology must first ensure the normal operation of the camera, but the existing algorithm lacks verification of the healthy working condition of the camera. On the other hand, drone image stitching technology has high real-time requirements. However, the existing feature extraction and image stitching technology has high computational complexity and lacks the necessary edge verification, resulting in the problem of overlapping areas. Therefore, it needs to be improved. Summary of the Invention

[0004] The present invention provides a real-time stitching method and system for unmanned aerial vehicle (UAV)-based forensic imagery to address the current technical problems of low computational efficiency and lack of necessary verification in image stitching.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In one aspect, the present invention provides a method for real-time stitching of drone-based forensic images, the method comprising:

[0007] The drone is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras.

[0008] Build a depth camera health detection algorithm to determine whether the depth camera is working properly. If it is working properly, perform image stitching. If it is not working properly, report the information to the command center.

[0009] If both depth cameras are working properly, obtain the primary and secondary images at the current moment, predict the boundaries of the overlapping areas, and verify the boundaries of the overlapping areas;

[0010] If the predicted overlapping area boundary is inconsistent with the verified overlapping area boundary, the random step algorithm is used to correct the overlapping area boundary to obtain the accurate overlapping area;

[0011] After obtaining the accurate overlapping area, initialize the blank image to complete the image stitching task;

[0012] Furthermore, the drone is equipped with two depth cameras and a GPS signal collector, the depth cameras collect environmental information, and the initialization of the depth cameras is completed, including:

[0013] The drone is equipped with two depth cameras, located on both sides of the drone's head. The center points of the two depth cameras are at the same horizontal position, and the image size captured by the two depth cameras is 480*640.

[0014] The two depth cameras are divided into a main depth camera and an auxiliary depth camera, which respectively collect environmental information and obtain two depth images, namely the main image F 1 ={R 1 , G 1 , B 1 , H 1} and auxiliary image F 2 ={R 2 , G 2 , B 2 , H 2}, where H 1 Represents the depth information of each pixel in the main image, R 1 , G 1 , B 1 Represents the color information of each pixel in the main image, H 2 Represents the depth information of each pixel in the auxiliary image, R 2 , G 2 , B 2 Represents the color information of each pixel in the auxiliary image;

[0015] Complete the color normalization processing of the main image and the auxiliary image. The specific formula of color normalization is shown as follows:

[0016] S 1 =0.265*R 1 +0.356*G 1 +0.256*B 1

[0017] S 2 =0.265*R 2 +0.356*G 2 +0.256*B 2

[0018] Among them, S 1 Indicates the grayscale value of each pixel in the main image after normalization, S 2 Represents the grayscale value of each pixel in the auxiliary image after normalization;

[0019] In the subsequent image stitching process, the calculation complexity of using three primary colors is high, so grayscale values ​​are used for calculation. For the convenience of subsequent description, the main image and auxiliary image captured by the main depth camera and the auxiliary depth camera are both images containing grayscale values ​​after color normalization.

[0020] Furthermore, the depth camera health detection algorithm is constructed to determine whether the depth camera is working properly. If it is working properly, image stitching is performed; if it is not working properly, the information is reported to the headquarters, including:

[0021] Building a depth camera health detection algorithm mainly includes the following steps:

[0022] (1) The main depth camera captures the main image at the 180th second and the main image at 179 seconds The auxiliary depth camera collects auxiliary images at the 180th second. and the auxiliary image at 179 seconds The grayscale values ​​of the pixels in the main image and the auxiliary image are used to calculate the drone position obtained by the main depth camera at the 180th second through the inter-frame matching algorithm. And the drone position obtained by the auxiliary depth camera at 180 seconds

[0023] (2) The GPS signal collector obtains the drone's position at the 180th second.

[0024] (3) Initialize the health detection threshold μ;

[0025] (4) Calculate the health coefficients γ1 and γ2 of the depth camera. The specific expressions are as follows:

[0026]

[0027] Among them, || represents the absolute value;

[0028] (5) If γ1≤μ and γ2≤μ, it indicates that both the main depth camera and the auxiliary depth camera are working normally, otherwise the main depth camera or the auxiliary depth camera is faulty;

[0029] (6) If both the main depth camera and the auxiliary depth camera are found to be working properly, perform image stitching to complete the evidence collection task. If the main depth camera or the auxiliary depth camera is faulty, report it to the command center for timely repair and replacement.

[0030] To ensure the accuracy of drone evidence collection, it is necessary to ensure that the depth camera is working properly throughout the entire evidence collection cycle. Therefore, after the initial completion of the depth camera health check, another camera health check is performed every 10 minutes.

[0031] Furthermore, if the depth cameras are all operating normally, obtaining the primary image and the auxiliary image at the current moment, predicting the boundary of the overlapping area, and verifying the boundary of the overlapping area include:

[0032] If the depth cameras are working properly, get the main image at the current moment and auxiliary images Where i represents an integer;

[0033] Predict the boundary of the overlapping area. Since the positions of the main depth camera and the auxiliary depth camera are fixed, the same horizontal angle δ1 of the main depth camera and the auxiliary depth camera and the shooting angle sizes δ2 and δ3 in the horizontal direction of the main depth camera and the auxiliary depth camera can be obtained.

[0034] The overlap ratio ρ is calculated by the same horizontal angle δ1 of the main depth camera and the auxiliary depth camera and the horizontal shooting angles δ2 and δ3 of the main depth camera and the auxiliary depth camera. The specific expression is shown in the following formula:

[0035] ρ=|δ2+δ3-2*δ1| / (δ2+δ3)

[0036] The main image at the current moment and auxiliary images The size of is 480*640, so the size of the overlapping area is ρ*640*480. In the process of drone evidence collection, the width of the overlapping area is consistent with 480, so the length of the overlapping area is ρ*640;

[0037] So the left boundary of the overlapping area can be predicted and right border The specific expression is as follows:

[0038]

[0039] Where pre means prediction, Zuo means left, You means right, and L means boundary. Indicates that the left boundary of the overlapping area corresponds to the boundary in the main image, Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image, Indicates that the right boundary of the overlapping area corresponds to the boundary in the auxiliary image;

[0040] The boundaries of the predicted overlapping areas are then verified;

[0041] Due to the installation position and collision during use, the horizontal angle between the main depth camera and the auxiliary depth camera and the horizontal shooting angle between the main depth camera and the auxiliary depth camera may change. If the image is stitched directly according to the boundary of the predicted overlapping area, it is easy to cause artifacts. Therefore, the boundary of the predicted overlapping area needs to be verified;

[0042] The verification steps are as follows:

[0043] (1) Calculate the left boundary of the overlapping area in the predicted overlapping area boundary corresponding to the boundary in the main image The left edge of the overlapping area corresponds to the edge in the auxiliary image The cumulative depth error and cumulative grayscale error

[0044]

[0045] The gray value of the point, i represents an integer;

[0046] (2) Calculate the boundary of the right edge of the overlapping area in the predicted overlapping area, which corresponds to the boundary in the main image The left edge of the overlapping area corresponds to the edge in the auxiliary image The cumulative depth error and cumulative grayscale error

[0047]

[0048] in, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image The depth value of the i-th pixel in , Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image The depth value of the i-th pixel, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image The gray value of the i-th pixel in , Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image Gray value of the i-th pixel;

[0049] (3) Initialize the cumulative depth threshold and cumulative grayscale threshold

[0050] (4) Initialize the verification flag FLAG and compare and Differences and comparisons and The specific expression is as follows:

[0051]

[0052] Among them, ∧ represents and, when and At the same time, FLAG=1, when and At the same time, FLAG = 1, ∨ means or, when or Only one is true, FLAG=0, when and Only one is true, FLAG=0;

[0053] Output the verification flag result. If the verification flag result is (1,1), it means that the predicted boundary of the overlapping area is accurate and the image stitching is performed directly. Otherwise, the predicted boundary of the overlapping area is inconsistent with the verified boundary of the overlapping area, and further correction is required to obtain the accurate boundary of the overlapping area.

[0054] Furthermore, if the predicted overlapping region boundary is inconsistent with the verified overlapping region boundary, a random step algorithm is used to correct the overlapping region boundary to obtain an accurate overlapping region, including:

[0055] If the predicted overlapping region boundary is inconsistent with the verified overlapping region boundary, the random step algorithm is used to correct the overlapping region boundary;

[0056] In the predicted overlapping area, the left boundary of the overlapping area corresponds to the boundary in the main image. and the boundaries in the auxiliary image The right edge of the overlapping area corresponds to the edge in the main image and the boundaries in the auxiliary image The boundaries in the auxiliary image and the boundaries in the main image is accurate, so it is called an accurate boundary, while the boundary in the main image and the boundaries in the auxiliary image It is inaccurate, so it is called an inaccurate boundary. Therefore, the purpose of correction is to find the boundary in the main image. and the boundaries in the auxiliary image Wherein, m and n are integers;

[0057] The specific steps of the random step algorithm are as follows:

[0058] (1) Get the current time stamp t and calculate the random step number O = (t%3) + 2;

[0059] (2) If O is an odd number, select O columns from the left side of the corresponding uncertain boundary, calculate the depth cumulative error between each column and the corresponding accurate boundary, and mark the selected columns;

[0060] (3) When calculating the depth cumulative error, only the pixels whose row coordinates are divisible by 5 are selected for calculation, and a total of 96 pixels are selected;

[0061] (4) Initialize the correction threshold τ. If the depth cumulative error between a column and the corresponding accurate boundary is less than the correction threshold τ, then the column is the boundary to be searched for correction. Otherwise, calculate the depth cumulative error of the next column.

[0062] (5) When the search for O columns on the left side is completed and no boundary that meets the conditions is found, it is necessary to continuously select O columns from the right side of the corresponding uncertain boundary and calculate again;

[0063] (6) After searching O columns on the left and right, if the column is not found, select the unmarked columns next to the marked columns and calculate until a column that meets the conditions is found. Otherwise, skip the frame and alarm.

[0064] Furthermore, after obtaining the accurate overlapping area, a blank image is initialized to complete the image stitching task, including:

[0065] After obtaining the accurate overlapping area, calculate the left boundary of the accurate overlapping area and right border The specific expression is as follows:

[0066]

[0067] Among them, n+m=639, true means accurate;

[0068] Initialize a blank image with a size of 480*(1280-n-1);

[0069] right To perform distortion compensation, the specific steps are as follows:

[0070] (1) Calculation and Average grayscale value of each column and

[0071] (2) Calculate the distortion coefficient ω. The specific expression is shown as follows:

[0072]

[0073] (3) If ω<0.1, set the compensation thresholds φ1=0.6, φ2=0.4; if ω≥0.1, set the compensation thresholds φ1=0.8, φ2=0.2;

[0074] (4) Compensation The specific expression is as follows:

[0075]

[0076] Select the pixels from the nth column to the 639th column in all the main images and auxiliary images for stitching to complete the image stitching task.

[0077] On the other hand, the present invention also provides a real-time stitching system for drone-based forensic images, and the real-time stitching method system for drone-based forensic images includes:

[0078] Image acquisition module: The drone is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras.

[0079] Depth camera detection module: Build a depth camera health detection algorithm to determine whether the depth camera is working properly. If it is working properly, image stitching is performed. If it is not working properly, the information is reported to the command center.

[0080] Prediction and verification module: If the depth cameras are working properly, obtain the main image and auxiliary image at the current moment, predict the boundaries of the overlapping area, and verify the boundaries of the overlapping area;

[0081] Correction and stitching module: If the predicted overlapping area boundary is inconsistent with the verified overlapping area boundary, the random shift algorithm is used to correct the overlapping area boundary to obtain the accurate overlapping area; after obtaining the accurate overlapping area, the blank image is initialized to complete the image stitching task.

[0082] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0083] 1. When implementing drone evidence collection, the present invention mechanically fixes two depth cameras and places the center points of the two depth cameras at the same horizontal position, thereby reducing the computational complexity of image stitching during drone evidence collection. At the same time, a depth camera health detection algorithm is constructed to ensure that the drone operates normally during the evidence collection process, thereby ensuring that the drone obtains accurate evidence information. Compared with algorithms that lack depth camera health detection, this algorithm can promptly detect depth camera failures in the drone during the evidence collection process, avoiding inaccurate evidence collection due to depth camera failures, thereby reducing the computing power loss of image stitching and improving the risk resistance of the drone evidence collection process.

[0084] 2. When realizing the real-time stitching of drone forensic images, the present invention introduces the idea of ​​control theory, first predicts and then verifies. During the prediction, only the angle relationship is used to obtain the area of ​​the overlapping area, and the obtained overlapping area is further verified. Compared with other solutions, this solution completes the verification of the overlapping area boundary through grayscale value and depth value, reducing the amount of calculation, thereby improving the real-time performance of drone forensic image stitching and having higher efficiency.

[0085] 3. The present invention adds a correction link for the boundary of the overlapping area, corrects the boundary of the overlapping area through a random step algorithm, and obtains an accurate overlapping area. When correcting and calculating the cumulative depth error, the information of some pixel points is evenly extracted for calculation, which further improves the calculation efficiency. When stitching images, the influence of edge distortion is timely eliminated through distortion compensation, so that the acquired stitched image is smoother. Compared with the overlapping area obtained by the alignment method, the overlapping area can be found without extracting complex feature points in this solution, and the calculation is simple and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0087] Figure 1 A diagram illustrating the implementation of a method and system steps for real-time stitching of drone-based forensic images provided in an embodiment of the present invention;

[0088] Figure 2 A schematic diagram of a real-time stitching method and system modules based on drone-based forensic images provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0090] Example 1

[0091] This embodiment will describe in more detail a method for real-time stitching of drone-based forensic images in conjunction with the corresponding drawings. Figure 1As shown in FIG, the method is mainly divided into five steps. The first step is S1: the UAV is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras. The second step is to build a depth camera health detection algorithm to determine whether the depth cameras are working normally. If they are working normally, image stitching is performed. If they are not working properly, the information is reported to the command center. The third step is to obtain the main image and auxiliary image at the current moment if the depth cameras are working normally, predict the boundaries of the overlapping area, and verify the boundaries of the overlapping area. The fourth step is to correct the boundaries of the overlapping area by using a random step algorithm if the predicted overlapping area boundary is inconsistent with the verified overlapping area boundary to obtain the accurate overlapping area. The fifth step is to initialize the blank image after obtaining the accurate overlapping area to complete the image stitching task.

[0092] Specifically, the method of this embodiment includes the following steps:

[0093] Step S1: The drone is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras.

[0094] The drone is equipped with two depth cameras, located on both sides of the drone's head. The center points of the two depth cameras are at the same horizontal position, and the image size captured by the two depth cameras is 480*640.

[0095] The two depth cameras are divided into a main depth camera and an auxiliary depth camera, which respectively collect environmental information and obtain two depth images, namely the main image F 1 ={R 1 , G 1 , B 1 , H 1} and auxiliary image F 2 ={R 2 , G 2 , B 2 , H 2}, where H 1 Represents the depth information of each pixel in the main image, R 1 , G 1 , B 1 Represents the color information of each pixel in the main image, H 2 Represents the depth information of each pixel in the auxiliary image, R 2 , G 2 , B 2 Represents the color information of each pixel in the auxiliary image;

[0096] Complete the color normalization processing of the main image and the auxiliary image. The specific formula of color normalization is shown as follows:

[0097] S 1 =0.265*R1 +0.356*G 1 +0.256*B 1

[0098] S 2 =0.265*R 2 +0.356*G 2 +0.256*B 2

[0099] Among them, S 1 Indicates the grayscale value of each pixel in the main image after normalization, S 2 Represents the grayscale value of each pixel in the auxiliary image after normalization;

[0100] In the subsequent image stitching process, the calculation complexity of using three primary colors is high, so grayscale values ​​are used for calculation. For the convenience of subsequent description, the main image and auxiliary image captured by the main depth camera and the auxiliary depth camera are both images containing grayscale values ​​after color normalization.

[0101] Step S2: Build a depth camera health detection algorithm to determine whether the depth camera is working properly. If it is working properly, perform image stitching. If it is not working properly, report the information to the command center.

[0102] Building a depth camera health detection algorithm mainly includes the following steps:

[0103] (1) The main depth camera captures the main image at the 180th second and the main image at 179 seconds The auxiliary depth camera collects auxiliary images at the 180th second. and the auxiliary image at 179 seconds The grayscale values ​​of the pixels in the main image and the auxiliary image are used to calculate the drone position obtained by the main depth camera at the 180th second through the inter-frame matching algorithm. And the drone position obtained by the auxiliary depth camera at 180 seconds

[0104] (2) The GPS signal collector obtains the drone's position at the 180th second.

[0105] (3) Initialize the health detection threshold μ;

[0106] (4) Calculate the health coefficients γ1 and γ2 of the depth camera. The specific expressions are as follows:

[0107]

[0108] Among them, || represents the absolute value. In actual situations, the default The value of is accurate;

[0109] (5) If γ1≤μ and γ2≤μ, it indicates that both the main depth camera and the auxiliary depth camera are working normally, otherwise the main depth camera or the auxiliary depth camera is faulty;

[0110] (6) If both the main depth camera and the auxiliary depth camera are found to be working properly, perform image stitching to complete the evidence collection task. If the main depth camera or the auxiliary depth camera is faulty, report it to the command center for timely repair and replacement.

[0111] To ensure the accuracy of drone evidence collection, it is necessary to ensure that the depth camera is working properly throughout the entire evidence collection cycle. Therefore, after the initial completion of the depth camera health check, another camera health check is performed every 10 minutes.

[0112] Step S3: If both depth cameras are working properly, obtain the primary image and the secondary image at the current moment, predict the boundary of the overlapping area, and verify the boundary of the overlapping area, including:

[0113] If the depth cameras are working properly, get the main image at the current moment and auxiliary images Where i represents an integer;

[0114] Predict the boundary of the overlapping area. Since the positions of the main depth camera and the auxiliary depth camera are fixed, the same horizontal angle δ1 of the main depth camera and the auxiliary depth camera and the shooting angle size δ2 and δ3 of the main depth camera and the auxiliary depth camera in the horizontal direction can be obtained. Due to mechanical looseness and other reasons, the size of δ1 will change, thereby affecting the effect of image stitching;

[0115] The overlap ratio ρ is calculated by the same horizontal angle δ1 of the main depth camera and the auxiliary depth camera and the horizontal shooting angles δ2 and δ3 of the main depth camera and the auxiliary depth camera. The specific expression is shown in the following formula:

[0116] ρ=|δ2+δ3-2*δ1| / (δ2+δ3)

[0117] The main image at the current moment and auxiliary images The size of is 480*640, so the overlap area size is ρ*480*640. In the drone evidence collection process, the width of the overlap area is always 480, so the length of the overlap area is ρ*640. If ρ*640 is not an integer, it is rounded off.

[0118] So the left boundary of the overlapping area can be predicted and right border The specific expression is as follows:

[0119]

[0120] Where pre means prediction, Zuo means left, You means right, and L means boundary. Indicates that the left boundary of the overlapping area corresponds to the boundary in the main image, Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image, Indicates that the right boundary of the overlapping area corresponds to the boundary in the auxiliary image;

[0121] The boundaries of the predicted overlapping areas are then verified;

[0122] Due to the installation position and collision during use, the horizontal angle between the main depth camera and the auxiliary depth camera and the horizontal shooting angle between the main depth camera and the auxiliary depth camera may change. If the image is stitched directly according to the boundary of the predicted overlapping area, it is easy to cause artifacts. Therefore, the boundary of the predicted overlapping area needs to be verified;

[0123] The verification steps are as follows:

[0124] (1) Calculate the left boundary of the overlapping area in the predicted overlapping area boundary corresponding to the boundary in the main image The left edge of the overlapping area corresponds to the edge in the auxiliary image The cumulative depth error and cumulative grayscale error

[0125]

[0126] The gray value of the point, i represents an integer;

[0127] (2) Calculate the boundary of the right edge of the overlapping area in the predicted overlapping area, which corresponds to the boundary in the main image The left edge of the overlapping area corresponds to the edge in the auxiliary image The cumulative depth error and cumulative grayscale error

[0128]

[0129] in, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image The depth value of the i-th pixel in , Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image The depth value of the i-th pixel, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image The gray value of the i-th pixel in , Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image Gray value of the i-th pixel;

[0130] (3) Initialize the cumulative depth threshold and cumulative grayscale threshold

[0131] (4) Initialize the verification flag FLAG and compare and Differences and comparisons and The specific expression is as follows:

[0132]

[0133] Among them, ∧ represents and, when and At the same time, FLAG=1, when and At the same time, FLAG = 1, ∨ means or, when or Only one is true, FLAG=0, when and Only one is true, FLAG=0;

[0134] Output the verification flag result. If the verification flag result is (1,1), it means that the predicted boundary of the overlapping area is accurate and the image stitching is performed directly. Otherwise, the predicted boundary of the overlapping area is inconsistent with the verified boundary of the overlapping area, and further correction is required to obtain the accurate boundary of the overlapping area.

[0135] Step S4: If the predicted overlapping region boundary is inconsistent with the verified overlapping region boundary, the random step algorithm is used to correct the overlapping region boundary to obtain an accurate overlapping region;

[0136] If the predicted overlapping region boundary is inconsistent with the verified overlapping region boundary, the random step algorithm is used to correct the overlapping region boundary;

[0137] In the predicted overlapping area, the left boundary of the overlapping area corresponds to the boundary in the main image. and the boundaries in the auxiliary image The right edge of the overlapping area corresponds to the edge in the main image and the boundaries in the auxiliary image The boundaries in the auxiliary image and the boundaries in the main image is accurate, so it is called an accurate boundary, while the boundary in the main image and the boundaries in the auxiliary image It is inaccurate, so it is called an inaccurate boundary. Therefore, the purpose of correction is to find the boundary in the main image. and the boundaries in the auxiliary image Wherein, m and n are integers;

[0138] The specific steps of the random step algorithm are as follows:

[0139] (1) Get the current time stamp t and calculate the random step number O = t% 3 + 2;

[0140] (2) If O is an odd number, select O columns from the left side of the corresponding uncertain boundary, calculate the depth cumulative error between each column and the corresponding accurate boundary, and mark the selected columns;

[0141] (3) When calculating the depth cumulative error, only the pixels whose row coordinates are divisible by 5 are selected for calculation, and a total of 96 pixels are selected;

[0142] (4) Initialize the correction threshold τ. If the depth cumulative error between a column and the corresponding accurate boundary is less than the correction threshold τ, then the column is the boundary to be searched for correction. Otherwise, calculate the depth cumulative error of the next column.

[0143] (5) When the search for O columns on the left side is completed and no boundary that meets the conditions is found, it is necessary to continuously select O columns from the right side of the corresponding uncertain boundary and calculate again;

[0144] (6) After searching O columns on the left and right, if the column is not found, select the unmarked columns next to the marked columns and calculate until a column that meets the conditions is found. Otherwise, skip the frame and alarm.

[0145] Step S5: After obtaining the accurate overlapping area, initialize the blank image to complete the image stitching task;

[0146] After obtaining the accurate overlapping area, the left boundary of the accurate overlapping area can be obtained. and right border The specific expression is as follows:

[0147]

[0148]

[0149] Among them, n+m=639, true means accurate;

[0150] Initialize a blank image with a size of 480*(1280-n-1);

[0151] right To perform distortion compensation, the specific steps are as follows:

[0152] (1) Calculation and Average grayscale value of each column and

[0153] (2) Calculate the distortion coefficient ω. The specific expression is shown as follows:

[0154]

[0155] (3) If ω<0.1, set the compensation thresholds φ1=0.6, φ2=0.4; if ω≥0.1, set the compensation thresholds φ1=0.8, φ2=0.2;

[0156] (4) Compensation The specific expression is as follows:

[0157]

[0158] Pixels from columns n to 639 in all the main and auxiliary images are selected for stitching to complete the image stitching task. In this solution, there is at least one column of overlapping area by default.

[0159] Example 2

[0160] This embodiment provides a real-time stitching system for drone-based forensic images. The main modules of the real-time stitching system for drone-based forensic images are as follows: Figure 2 As shown:

[0161] Image acquisition module: The drone is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras.

[0162] Depth camera detection module: Build a depth camera health detection algorithm to determine whether the depth camera is working properly. If it is working properly, image stitching is performed. If it is not working properly, the information is reported to the command center.

[0163] Prediction and verification module: If the depth cameras are working properly, obtain the main image and auxiliary image at the current moment, predict the boundaries of the overlapping area, and verify the boundaries of the overlapping area;

[0164] Correction and stitching module: If the predicted overlapping area boundary is inconsistent with the verified overlapping area boundary, the random shift algorithm is used to correct the overlapping area boundary to obtain the accurate overlapping area; after obtaining the accurate overlapping area, the blank image is initialized to complete the image stitching task.

[0165] The real-time stitching system for drone forensic images of this embodiment corresponds to the real-time stitching method for drone forensic images of the above embodiment; wherein, the functions implemented by each functional module in the real-time stitching system for drone forensic images of this embodiment correspond one-to-one to each process step in the real-time stitching method for drone forensic images of the above embodiment; therefore, they will not be repeated here.

[0166] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments. In addition, the terms "including", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method comprising the element.

[0167] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0168] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time stitching method for drone-based forensic images, characterized in that: The following steps are involved: S1: The drone is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras. The two depth cameras are located on both sides of the drone's head and are divided into a main depth camera and an auxiliary depth camera. S2: Build a depth camera health detection algorithm to determine whether the depth camera is working properly. If it is working properly, perform image stitching. If it is not working properly, report the information to the command center. S3: If both depth cameras are working properly, obtain the primary image and auxiliary image at the current moment, predict the boundary of the overlapping area, and verify the boundary of the overlapping area; including: If the depth cameras are working properly, get the main image at the current moment and auxiliary images Where i is an integer; Predict the boundary of the overlapping area. Since the positions of the main depth camera and the auxiliary depth camera are fixed, the same horizontal angle δ1 of the main depth camera and the auxiliary depth camera and the horizontal shooting angles δ2 and δ3 of the main depth camera and the auxiliary depth camera are calculated; The overlap ratio ρ is calculated by the same horizontal angle δ1 of the main depth camera and the auxiliary depth camera and the horizontal shooting angles δ2 and δ3 of the main depth camera and the auxiliary depth camera. The specific expression is shown in the following formula: ρ=|δ2+δ3-2*δ1| / (δ2+δ3) The main image at the current moment and auxiliary images The size of is 480*640, so the size of the overlapping area is ρ*640*480. In the process of drone evidence collection, the width of the overlapping area is consistent with 480, so the length of the overlapping area is ρ*640; Predict the left boundary of the overlapping area and right border The specific expression is as follows: Among them, pre means prediction, Zuo means left, You means right, L means boundary, L 1 (1-ρ)*639 Indicates that the left boundary of the overlapping area corresponds to the boundary in the main image, Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image, Indicates that the right boundary of the overlapping area corresponds to the boundary in the auxiliary image; The boundaries of the predicted overlapping areas are then verified; The verifying the boundary of the predicted overlapping area includes: The verification steps are as follows: (1) Calculate the boundary L in the main image corresponding to the left boundary of the overlapping area in the predicted overlapping area boundary 1 (1-ρ)*639 The left edge of the overlapping area corresponds to the edge in the auxiliary image The cumulative depth error and cumulative grayscale error in, Indicates that the left boundary of the overlapping area corresponds to the boundary L in the main image 1 (1-ρ)*639 No. i The gray value of pixel i, Indicates that the left boundary of the overlapping area corresponds to the boundary in the auxiliary image The grayscale value of the i-th pixel, where i represents an integer; (2) Calculate the boundary of the right edge of the overlapping area in the predicted overlapping area, which corresponds to the boundary in the main image The right edge of the overlapping area corresponds to the edge in the auxiliary image The cumulative depth error and cumulative grayscale error in, Indicates that the right boundary of the overlapping area corresponds to the boundary in the main image The i-th pixel in value, Indicates that the right boundary of the overlapping area corresponds to the boundary in the auxiliary image Gray value of the i-th pixel; (3) Initialize the cumulative depth threshold and cumulative grayscale threshold (4) Initialize the verification flag FLAG and compare and Differences and comparisons and The specific expression is as follows: Among them, ∧ represents and, when and At the same time, FLAG=1, when and At the same time, FLAG = 1, ∨ means or, when or Only one is true, FLAG=0, when or Only one is true, FLAG=0; Output the verification flag result. If the verification flag result is (1,1), it means that the predicted overlapping area boundary is accurate and the image stitching is performed directly. Otherwise, the predicted overlapping area boundary and the verified overlapping area boundary are inconsistent and further correction is required to obtain the accurate overlapping area boundary. S4: If the predicted overlapping region boundary is inconsistent with the verified overlapping region boundary, the random shift algorithm is used to correct the overlapping region boundary to obtain the accurate overlapping region; S5: After obtaining the accurate overlapping area, initialize the blank image to complete the image stitching task.

2. The real-time stitching method for drone-based forensic images according to claim 1, characterized in that: The step S1 comprises: The center points of the two depth cameras are located at the same horizontal position, and the image size of the two depth cameras is 480*640. They collect environmental information respectively and obtain two depth images, namely the main image F 1 ={R 1 , G 1 , B 1 , H 1 } and auxiliary image F 2 ={R 2 , G 2 , B 2 , H 2 }, where H 1 Represents the depth information of each pixel in the main image, R 1 , G 1 , B 1 Represents the color information of each pixel in the main image, H 2 Represents the depth information of each pixel in the auxiliary image, R 2 , G 2 , B 2 Represents the color information of each pixel in the auxiliary image; Complete the color normalization processing of the main image and the auxiliary image. The specific formula of color normalization is shown as follows: S 1 =0.298*R 1 +0.578*G 1 +0.134*B 1 S 2 =0.298*R 2 +0.578*G 2 +0.134*B 2 Among them, S 1 Indicates the grayscale value of each pixel in the main image after normalization, S 2 Indicates the grayscale value of each pixel in the auxiliary image after normalization.

3. The real-time stitching method for drone-based forensic images according to claim 1, characterized in that: The step S2 comprises: Building a depth camera health detection algorithm involves the following steps: (1) The main depth camera captures the main image at the 180th second and the main image at 179 seconds The auxiliary depth camera collects auxiliary images at the 180th second. and the auxiliary image at 179 seconds The grayscale values ​​of the pixels in the main image and the auxiliary image are used to calculate the drone position obtained by the main depth camera at the 180th second through the inter-frame matching algorithm. And the drone position obtained by the auxiliary depth camera at 180 seconds (2) The GPS signal collector obtains the drone's position at the 180th second. (3) Initialize the health detection threshold μ; (4) Calculate the health coefficients y1 and y2 of the depth camera. The specific expressions are as follows: Among them, || represents the absolute value; (5) If y1≤μ and y2≤μ, it indicates that both the main depth camera and the auxiliary depth camera are working properly, otherwise the main depth camera or the auxiliary depth camera is faulty; (6) If it is found that the main depth camera and the auxiliary depth camera are working normally, the image stitching is performed to complete the evidence collection task. If there is a fault in the main depth camera or the auxiliary depth camera, it is reported to the command center for timely inspection and replacement.

4. The real-time stitching method for drone-based forensic images according to claim 1, characterized in that: The step S4 comprises: If the predicted overlapping region boundary is inconsistent with the verified overlapping region boundary, the random step algorithm is used to correct the overlapping region boundary; The specific steps of the random step algorithm are as follows: (1) Get the current time stamp t and calculate the random step number O = t%3 + 2, where % represents the remainder operation; (2) If O is an odd number, select O columns from the left side of the corresponding uncertain boundary, calculate the depth cumulative error between each column and the corresponding accurate boundary, and mark the selected columns; (3) When calculating the depth cumulative error, only the pixels whose row coordinates are divisible by 5 are selected for calculation, and a total of 96 pixels are selected; (4) Initialize the correction threshold τ. If the depth cumulative error between a column and the corresponding accurate boundary is less than the correction threshold τ, then the column is the boundary to be searched for correction. Otherwise, the calculation of the depth cumulative error of the next column is performed. (5) When the search for O columns on the left side is completed and no boundary that meets the conditions is found, it is necessary to continuously select O columns from the right side of the corresponding uncertain boundary and calculate again; (6) After searching O columns on the left and right, if the column is not found, select the unmarked columns next to the marked columns for calculation until a column that meets the conditions is found. Otherwise, skip the current frame and alarm.

5. The real-time stitching method for drone-based forensic images according to claim 1, characterized in that: Place The step S5 includes: After obtaining the exact overlapping area, calculate the left boundary of the exact overlapping area and the exact right edge of the overlapping area The specific expression is as follows: Among them, n+m=639, true means accurate; Initialize a blank image with a size of 480*(1280-n-1); right To perform distortion compensation, the specific steps are as follows: (1) Calculation and Average grayscale value of each column and (2) Calculate the distortion coefficient ω. The specific expression is as follows: (3) If ω<0.1, set the compensation thresholds φ1=0.6, φ2=0.4; if ω≥0.1, set the compensation thresholds φ1=0.8, φ2=0.2; (4) Compensation The specific expression is as follows: Select the pixels from the nth column to the 639th column in all the main images and auxiliary images for stitching to complete the image stitching task.

6. A real-time stitching system based on drone-based forensic images, characterized in that: include: Image acquisition module: The drone is equipped with two depth cameras and a GPS signal collector. The depth cameras collect environmental information and complete the initialization of the depth cameras. Depth camera detection module: Build a depth camera health detection algorithm to determine whether the depth camera is working properly. If it is working properly, image stitching is performed. If it is not working properly, the information is reported to the command center. Prediction and verification module: If the depth cameras are working properly, obtain the main image and auxiliary image at the current moment, predict the boundaries of the overlapping area, and verify the boundaries of the overlapping area; Correction and stitching module: If the predicted overlapping area boundary is inconsistent with the verified overlapping area boundary, the random shift algorithm is used to correct the overlapping area boundary to obtain the accurate overlapping area. After obtaining the accurate overlapping area, the blank image is initialized to complete the image stitching task. Implement a real-time stitching method for drone-based forensic images as described in any one of claims 1-5.

Images