Multi-machine large-field fast cascade splicing system and method for weak texture background

Abstract

A multi-machine large-field fast cascade stitching system and method for weak-textured background, the method comprising acquiring the posture data of two machines to obtain the posture conversion matrix of the two machines; receiving the video data of the two machines and decoding them to obtain the grayscale images of the two machines; solving the overlapping ROI area of ​​the grayscale image; cropping the grayscale image to obtain the cropped image; performing SIFT feature point extraction; mapping the feature point set of the first flying device to the second flying device, performing rough matching to obtain the mapping point set mapped to the second flying device; selecting the feature point set in the feature point set of the second flying device and the neighborhood of the mapping point set, and performing precise matching between the mapping point set and the feature point set to obtain a feature point matching pair; calculating the homography matrix based on the feature point matching pair; and performing large-field stitching of the views of the two machines according to the homography matrix. The present invention has high stitching accuracy and is suitable for weak-textured backgrounds such as grasslands, wastelands, jungles, and waters.

Description

Technical Field

[0001] The invention relates to unmanned aerial vehicle video image stitching, and in particular to a multi-machine large-field-of-view fast cascade stitching system and method thereof with a weak-textured background. Background Art

[0002] With the rapid development of artificial intelligence and optoelectronic technology, drone aerial photography has important application value in surveying and mapping, security, search and rescue, inspection and other fields. Among them, the advantages of multi-drone collaboration in full coverage of the search area and large-scale inspection imaging are more obvious. Multiple drones carry high-definition visible light pods and perform aerial imaging of the environmental area according to the track planning. On this basis, the high-definition video frames of each drone are spliced ​​to form a large field of view image, which is conducive to the rapid search of ground objects.

[0003] At present, the algorithm idea for UAV video stitching is to first calculate the manual feature points of the two perspective images, and then align and match the feature points to calculate the homography matrix of the two views. The processing flow is usually in the form of "online collection + post-processing", that is, the drone first completes the image data acquisition and recording, and after completing the aerial flight, the recorded data is post-processed on a large server. The above method can meet the needs of most application scenarios, but there are still some shortcomings: on the one hand, there are still some shortcomings for some weak texture background stitching tasks where feature point extraction is difficult and there are many similar feature points. For example, grassland In search and rescue missions such as in wilderness, jungles, and waters, it is necessary to construct observation images with a large field of view, and existing methods often fail. On the other hand, the above methods are also difficult to meet for tasks such as surveying and mapping, search and rescue, which have high real-time requirements and require real-time online presentation of the background of the object with a large field of view. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned background technology and provide a multi-machine large field of view fast cascade stitching system and method for weak texture background, which has high stitching accuracy and is suitable for weak texture backgrounds such as grasslands, wastelands, jungles, and waters.

[0005] The technical solution adopted by the present invention to solve the technical problem is a multi-machine large field of view fast cascade splicing system with weak texture background, comprising: a first flying device, a second flying device and a ground display and control platform, a data computing platform is installed on the first flying device, the first flying device is wirelessly connected to the second flying device, and the first flying device and the second flying device are respectively connected to the ground display and control platform; the data computing platform comprises ARM and FPGA, ARM is connected to FPGA, and the first flight device and the second flight device are in master-slave following mode; the first flight device includes a first UAV, a first pod, a first airborne switch, a first flight control device, and a first airborne radio, and the first UAV is installed on the first pod; the data computing platform is connected to the first airborne switch, and the first airborne switch is respectively connected to the first pod, the first flight control device, and the first airborne radio; the second flight device includes a second UAV, a second pod, a second airborne switch, a second flight control device, and a second airborne radio, and the second UAV is installed on the second pod, and the second airborne switch is respectively connected to the second pod, the second flight control device, and the second airborne radio; the first airborne radio is wirelessly connected to the second airborne radio; the first airborne radio and the second airborne radio are respectively connected to the ground display and control platform; the first UAV and the second UAV are equipped with an inertial navigation unit and a GPS.

[0006] A multi-machine large-field fast cascade stitching method for weak texture background includes the following steps:

[0007] Step S1: ARM acquires the posture data of the first flying device and the second flying device in real time, and obtains the posture conversion matrix H of the two flying devices according to the posture data;

[0008] Step S2: ARM receives the video data of the first flying device and the second flying device in real time, and decodes them to obtain grayscale images of the two flying devices; solves the overlapping ROI area of ​​the grayscale images based on the posture conversion matrix H; crops the grayscale images of the two flying devices based on the overlapping ROI area to obtain cropped grayscale images R1 and R2 of the two flying devices; and transmits the cropped grayscale images R1 and R2 to FPGA;

[0009] Step S3: FPGA extracts SIFT feature points from the cropped grayscale images R1 and R2 of the two flying devices, obtains SIFT feature point sets T1 and T2 of the two flying devices, and transmits them to ARM;

[0010] Step S4: ARM maps the SIFT feature point set T1 of the first flying device to the second flying device based on the posture conversion matrix H of the two flying devices, performs rough matching, and obtains the mapping point set t1 mapped to the second flying device;

[0011] Step S5: Selecting a SIFT feature point set t2 in the SIFT feature point set T2 of the second flying device that is in the neighborhood of the mapping point set t1, and performing precise matching between the mapping point set t1 and the SIFT feature point set t2 to obtain a feature point matching pair;

[0012] Step S6: Calculate the homography matrix based on the feature point matching pairs, and send the homography matrix to the ground display and control platform;

[0013] Step S7: The ground display and control platform performs large-field stitching of the views of the two flight devices according to the homography matrix.

[0014] Furthermore, in step S1, the method for obtaining the posture conversion matrix H of the two flying devices is as follows:

[0015] First flight unit in position Take a picture of the ground scene and get an image , the second flying device is in position Take a picture of the ground scene and get an image ;

[0016] For a point in the scene, Indicates that M is in position The camera coordinates, Indicates that M is in position The camera coordinates of For M in the image The corresponding point on For M in the image The corresponding points on the graph are as follows: , then the matrix H is called the image arrive The pose transformation matrix is is a non-zero constant term; according to the imaging principle of the camera, the pose conversion matrix between two aerial images can be expressed as:

[0017] (1)

[0018] Where K is the intrinsic parameter matrix of the camera, and the first drone is at position and the second drone in position The movement between is R×[I,-t], where I represents The identity matrix of The rotation matrix is ​​t, t is the three-dimensional translation vector; n is the unit normal vector of the ground; d is the distance from the origin of the coordinate system to the ground plane;

[0019] The location where the drone was taken It is expressed by 6 parameters, denoted as , They represent the deflection angle, pitch angle, and roll angle of the drone relative to the true north. Respectively represent the relative coordinate origin of the drone x , y , z Axis coordinates, i=1 or 2, where the origin of the coordinates is a point with known longitude and latitude, with due north as y Axis, due east x Axis, perpendicular to xy The plane is vertically upward z axis;

[0020] according to , Determine R and t. The specific calculation rules are calculated according to Euler's formula:

[0021] (3)

[0022] (4)

[0023] in: , , The Euler angles for yaw, pitch, and roll, respectively.

[0024] Further, in step S2, the specific method for calculating the overlapping ROI area is:

[0025] S2-1: Extract and match the features of the entire starting frame to obtain the stitching result of the first frame and the overlapping areas of both frames; for the stitching of two images, is an interior point of image 1, is the interior point of image 2, and the set of interior point pairs is: ,in is the inlier pair of inlier 1 in image 1, is an interior point in image 1 n The inner point pairs of the two images are sorted to obtain the points at the upper left corner and the lower right corner of the overlapping area of ​​the two images, and the overlapping area is obtained;

[0026] in is the coordinate point of the upper left corner of the overlapping area, is the coordinate point of the lower right corner of the overlapping area;

[0027] S2-2: In the next frame, feature extraction and matching are performed in the obtained overlapping area, and a new spliced ​​image and overlapping area are obtained for use in the next frame;

[0028] S2-3: Iterate to obtain a stable overlapping region as the final overlapping RIO region.

[0029] Furthermore, in step S2, a multi-threaded processing mechanism is adopted to realize the decoding of the video data of the two machines, the acquisition of image frames, and the calculation of overlapping ROI areas and image cropping based on the posture of the drones, as follows: two threads are designed, thread 1 obtains the postures of the two drones and the pod parameters to calculate the relative posture transformation matrix; thread 2 obtains the video data for decoding; thread 1 caches the calculated posture transformation matrix in a global variable in real time for thread 2 to read.

[0030] Furthermore, in step S2, the viewing angles of the first pod and the second pod are consistent and are set to 90° for aerial photography. The videos shot by the first pod and the second pod are in H264 stream format. ARM uses the RK3588 chip with an MPP hard decoding module, which can complete real-time decoding of the H264 format code stream through programming.

[0031] Further, in step S3, the specific process of FPGA performing SIFT feature point extraction includes: first, using the Gaussian kernel function to perform Gaussian pyramid preprocessing on the cropped grayscale image, and obtaining images of different scales after processing; second, performing feature point detection and positioning on images of different scales; then, by calculating the gradient histogram of the midpoint in the neighborhood of the feature point, the gradient amplitude and direction of the feature point are statistically analyzed, the main direction of the feature point is assigned, the direction of the point with the maximum gradient is selected as the main direction of the feature point, the feature point in the main direction is selected as the main feature point, and the position, scale, and direction features of the main feature point are recorded; next, the gradient of the neighborhood points of the main feature point is extracted to form a feature descriptor; finally, the position, scale, and direction features of the main feature point and the feature descriptors of the neighborhood points of the main feature point are packaged together as a SIFT feature vector to form a SIFT feature point set.

[0032] Furthermore, in steps S4 and S5, the specific methods of rough matching and fine matching are:

[0033] For the feature point pairs of the first drone view Any feature point in , according to the image coordinates of the feature points , the pose transformation matrix calculates the image position corresponding to the second drone view , in the view of the second drone, with the coordinate point as the center, search for the most similar feature point in its 100×100 neighborhood , as the optimal matching point of the feature point of the first drone view, thereby obtaining a feature point matching pair of the first drone and the second drone view; all feature points of the first drone view are searched in the above manner to obtain all feature point matching pairs .

[0034] Furthermore, in step S6, a transformation matrix of the feature points is calculated based on the obtained feature point matching pairs as the homography matrix of the two-machine view mosaic.

[0035] Compared with the prior art, the advantages of the present invention are as follows:

[0036] In view of the low precision and high real-time requirements of large-field video splicing under weak texture background, and the difficulty in extracting feature points under weak texture background, the present invention first performs coarse matching based on the attitude mapping between drones, and then performs fine matching on the manual feature points in the neighborhood after coarse matching to obtain a robust single-image matrix. The large-field real-time splicing of the views of the two flying devices is performed according to the single-image matrix, with high splicing accuracy, and is suitable for weak texture backgrounds such as grasslands, wastelands, jungles, and waters. In addition, in order to achieve real-time online computing, the present invention deploys the algorithm to the ARM+FPGA edge computing platform carried by the drone, and realizes real-time computing requirements through computing acceleration of ARM and FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of a multi-machine large-field-of-view fast cascade stitching method for a weak-textured background according to an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the principle of multi-machine large-field fast cascade stitching of texture background in an embodiment of the present invention.

[0039] Figure 3 It is a schematic diagram of a video decoding process according to an embodiment of the present invention.

[0040] Figure 4 It is a schematic diagram of the SIFT feature extraction process of ARM and FPGA in an embodiment of the present invention.

[0041] Figure 5 It is a schematic diagram of the PCIe packet transmission principle of ARM and FPGA in an embodiment of the present invention.

[0042] Figure 6 It is a schematic diagram of the SIFT feature extraction process based on FPGA optimization in an embodiment of the present invention.

[0043] FIG7 is a real-time mosaic effect diagram viewed on the ground display and control platform of an embodiment of the present invention, wherein FIG7(a) is a mosaic effect diagram of wasteland, and FIG7(b) is a mosaic effect diagram of water area / forest. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] A multi-machine large field of view fast cascade splicing system with weak texture background includes a first flying device, a second flying device and a ground display and control platform. The first flying device is equipped with a data computing platform. The first flying device is wirelessly connected to the second flying device. The first flying device and the second flying device are respectively connected to the ground display and control platform. The data computing platform includes ARM and FPGA. The ARM and FPGA are connected by PCIe (Peripheral Component Interconnect Express) high-speed connection.

[0046] The first and second aircraft are in master-slave following mode. The first aircraft is the master or slave, and the second aircraft is the slave or master. All device IP addresses of the first and second aircraft are set to the same segment to ensure data sharing and intercommunication.

[0047] The first flight device includes a first UAV, a first pod, a first airborne switch, a first flight control device, and a first airborne radio. The first UAV is installed on the first pod; the data computing platform is connected to the first airborne switch, and the first airborne switch is respectively connected to the first pod, the first flight control device, and the first airborne radio; the second flight device includes a second UAV, a second pod, a second airborne switch, a second flight control device, and a second airborne radio. The second UAV is installed on the second pod, and the second airborne switch is respectively connected to the second pod, the second flight control device, and the second airborne radio; the first airborne radio is wirelessly connected to the second airborne radio. The first airborne radio and the second airborne radio are respectively connected to the ground display and control platform. The first UAV and the second UAV are equipped with an inertial navigation unit (INU) and a GPS.

[0048] The first pod and the second pod are of the same model, with the same video resolution (720P or 1080P) and camera focal length. The actual focal length can be set according to the flight altitude and ground resolution requirements, such as 1x, 3x, 5x, etc. The IPs of the first pod, the second pod, the first airborne radio, and the second airborne radio are set to the same network segment and can ping each other.

[0049] Before the drone takes off, set the video resolution and camera focal length of the first and second pods, and calculate the distance between the two drones based on the drone flight altitude, pod pixel size, focal length, and required overlap rate. Calculate the distance between the two drones based on the preset drone flight altitude to ensure that the video overlap rate of the two drones is greater than 40%.

[0050] Before the drones take off, the first and second drones are set to the master-slave following mode, and the track of the first drone is set on the ground station. The second drone automatically generates a track according to the preset track of the first drone and the distance between the two drones. To ensure flight safety, the first and second drones always maintain a height difference of about 5 meters.

[0051] Reference Figure 1 , 2 , a multi-machine large field of view fast cascade stitching method with weak texture background, comprising the following steps:

[0052] Step S1: ARM acquires the posture data of the first flying device and the second flying device in real time, and obtains the posture conversion matrix H of the two flying devices according to the posture data;

[0053] First flight unit in position Take a picture of the ground scene and get an image , the second flying device is in position Take a picture of the ground scene and get an image ;

[0054] For a point in the scene, Indicates that M is in position The camera coordinates, Indicates that M is in position The camera coordinates. For M in the image The corresponding point on For M in the image The corresponding points on the graph are as follows: , then the matrix H is called the image arrive The pose transformation matrix is is a non-zero constant term. According to the imaging principle of the camera, the pose conversion matrix between two aerial images can be expressed as:

[0055] (1)

[0056] Where K is the intrinsic parameter matrix of the camera, and the first drone is at position and the second drone in position The movement between is R×[I,-t], where I represents The identity matrix of The rotation matrix of the image is t, t is the three-dimensional translation vector, n is the unit normal vector of the ground, and d is the distance from the origin of the coordinate system to the ground plane. The pose transformation matrix between the aerial images is obtained.

[0057] The location where the drone was taken It can be expressed by 6 parameters, denoted as , They represent the deflection angle, pitch angle, and roll angle of the drone relative to the true north. Respectively represent the relative coordinate origin of the drone x , y , zAxis coordinates, i=1 or 2, where the origin of the coordinates can be selected from a point with known longitude and latitude, usually with due north as the y Axis, due east x Axis, perpendicular to xy The plane is vertically upward z Axis, where the drone obtains its own latitude and longitude according to the onboard GPS, and then calculates the coordinate value relative to the origin according to formula (2), in meters.

[0058] (2)

[0059] In the formula, is the latitude and longitude of the drone, where It is the latitude and longitude of the reference point, usually replaced by the latitude and longitude of the ground station. Usually it is 0, is 6378245 meters, is the calculated coordinate of the drone relative to the origin.

[0060] The drone is equipped with an inertial navigation unit (INU) and GPS, which can output the attitude angle and coordinates of the drone in real time. , R and t can be determined. The specific calculation rules are calculated according to Euler's formula:

[0061] (3)

[0062] (4)

[0063] in: , , The Euler angles for yaw, pitch, and roll, respectively.

[0064] Therefore, the pose transformation matrix between two aerial images can be calculated by 6 parameters. It can be seen from the formula that the derivation of the pose transformation matrix between aerial images requires the same depth of all points in the image, that is, to ensure that all scenes in the image under the viewpoint are in the same imaging depth, otherwise the pose transformation between images cannot be satisfied. In practical applications, it is not strictly required that all pixels have the same depth, which is also one of the main sources of image stitching errors.

[0065] The internal parameters of the optoelectronic pod used in this embodiment are:

[0066] Distortion coefficient:

[0067] According to the calculation, when the flight altitude is 100 meters, the two aircraft are flying parallel, the horizontal distance is 25 meters, the front and rear distance is 5 meters, and the height difference is 5 meters, the splicing posture conversion matrix of the two aircraft is:

[0068]

[0069] The image stitching method based on pose information only involves processing the pose information of the drone to obtain the pose transformation matrix for image stitching. Since there is no need to extract the feature information of the image, only the pose transformation of the image is performed. This method is very efficient and fast, and the processing time is generally in milliseconds. The pose rough matching method has a good matching effect on the flat ground taken from above, but the matching effect is limited for two images at different depths, such as ground buildings and targets.

[0070] Step S2: ARM receives the video data of the first flying device and the second flying device in real time, and decodes them to obtain grayscale images of the two flying devices; solves the overlapping ROI area of ​​the grayscale images based on the posture conversion matrix H; crops the grayscale images of the two flying devices based on the overlapping ROI area to obtain cropped grayscale images R1 and R2 of the two flying devices; and transmits the cropped grayscale images R1 and R2 to FPGA;

[0071] In previous work, feature point extraction is often performed on the entire image, but the final feature matching process only requires matching feature points in the overlapping area. Extracting and matching in the entire image not only increases the uncertainty of matching, but also wastes time and computing resources. When the system moves at a low speed, there is almost no change between frames.

[0072] During the puzzle process of the two drones, the relative posture, flight status, and pod parameters of the drones can be obtained in real time. Therefore, the overlapping area of ​​the drone's field of view can be obtained based on the relative posture, pod parameters, etc. In order to simplify the feature matching process, only the feature points of the images in the overlapping area of ​​the two drones can be extracted and matched.

[0073] In the specific implementation process of the present invention for overlapping ROI area calculation and image cropping based on multi-threading, this embodiment designs a multi-thread processing mechanism to realize the decoding of video data of two machines, image frame acquisition, and overlapping ROI area calculation and image cropping based on the posture of the drone. Specifically as follows: two threads are designed, thread 1 obtains the postures of two drones and pod parameters to calculate the relative posture conversion matrix; thread 2 obtains video data for decoding; thread 1 caches the calculated posture conversion matrix in real time in a global variable for thread 2 to read.

[0074] In thread 1, the ARM of the first UAV reads the flight control and pod attitude information of the second UAV through the wireless ad hoc network.

[0075] Thread 2 mainly calculates the overlapping ROI area based on the posture conversion matrix of the two machines. The design of the two threads is actually to meet the requirements that the rough matching matrix (posture conversion matrix) of the two machines and the fine matching matrix (homography matrix) of the feature points of the two machine views are as consistent as possible in time to improve the matching accuracy. In thread 2, when ZLMedia receives a complete H264 video stream frame, it responds to trigger the H264 data decoding event; when a complete image frame is decoded, it responds to trigger the image cropping event; in the image cropping event, the first processing function reads the conversion matrix obtained by thread 1 from the global variable, and obtains the overlapping area of ​​the two machine views based on the conversion matrix, and performs format conversion and grayscale conversion operations on the image data in the overlapping area to obtain the grayscale image of the overlapping area. In order to further improve the efficiency of grayscale image preprocessing, the present invention uses an RGA acceleration module to accelerate grayscale conversion, cropping and other operations.

[0076] The specific method for calculating the overlapping ROI area is:

[0077] 1) Perform full-image feature extraction and matching on the starting frame to obtain the stitching result of the first frame and the overlapping areas of both frames. is an interior point of image 1, is the inlier point of image 2, where the inlier point is selected by RANSAC in feature matching. The set of inlier point pairs that can be obtained is: ,in is the inlier pair of inlier 1 in image 1, is an interior point in image 1 n Sort the sets in the two images, get the points in the upper left corner and the lower right corner of the overlapping area in the two images, and get the overlapping area.

[0078] in is the coordinate point of the upper left corner of the overlapping area, It is the coordinate point of the lower right corner of the overlapping area.

[0079] 2) In the next frame, feature extraction and matching are performed only in the obtained overlapping area, and a new stitched image and overlapping area are obtained for use in the next frame.

[0080] 3) In the process of calculating the overlapping area, the overlapping area is constrained by combining the flight control information (GPS signal, altitude, camera angle, flight speed information). If the proportion of the feature area is too small, the feature is directly extracted in the original image, otherwise it is directly extracted in the overlapping area.

[0081] Finally, a stable overlapping area can be iterated as the final overlapping RIO area, and the computational efficiency can be optimized by extracting and matching features in this area.

[0082] In addition, the processing flow of thread 2 can be simplified based on the pre-set overlap area parameters. The specific operations are as follows: Assume that the drone is at a certain altitude. , the base focal length of the pod is , zoom factor , the pixel size is , the pod resolution is , then the image frame length is , the image width is , the proposed interval between the two aircraft routes (the actual length of the interval between the routes) is , then the width of the flight path interval between the images taken by the two aircraft can be calculated The calculation formula is: , from which the overlap of the two aircraft routes can be calculated as: Usually, the overlap between the two aircraft is set to no less than 40%, and then the distance between the UAVs, the focal length of the UAV pod and other parameters are set according to the UAV altitude. Of course, in order to make the parameters more accurate, real-time calculation can be used to obtain the overlap area parameters.

[0083] In step S2, the first pod and the second pod have the same viewing angle, which is set to 90° for aerial photography. The videos shot by the first pod and the second pod are in H264 stream format. ARM uses the RK3588 chip with an MPP hard decoding module, which can complete real-time decoding of the H264 format code stream through programming.

[0084] The drone pod data is generally encoded in H264 format and transmitted over the network. Before video splicing, the H264 format video data needs to be decoded to obtain the original image frame data. This embodiment uses ZLMedia to obtain H264 video data, and then decodes the two-way video in real time based on the MPP hard decoding module of RK3588. RK3588 supports up to 32-channel video decoding and a maximum decoding performance of 8k / 60fps, which meets the system requirements of the design. As shown in Table 1, the delay comparison of various decoding methods is shown. Under normal circumstances, the soft decoding delay of H264 is about 1s. Compared with soft decoding, the delay of hard decoding is greatly reduced. The delay comparison of various decoding methods is shown in Table 1. It can be seen that the time of hard decoding is only 1.3% of that of soft decoding.

[0085]

[0086] ARM's decoding process for two-channel video data is as follows Figure 3As shown, it includes initializing two decoders, registering decoding processing functions, grayscale conversion, data packaging, etc. ARM obtains the H264 format video stream data of the first drone and the second drone through the RTSP stream addresses of the first pod and the second pod; ARM registers two decoding threads, namely the first decoder and the second decoder, respectively in a multi-threaded manner. The first decoder processes the video stream data of the first drone, and the second decoder processes the video stream data of the second drone. ARM decodes the H264 video stream according to the event mechanism, uses the ZLMedia library to read the RTSP stream address, and when a complete stream data packet is obtained, responds to MPP hard decoding to obtain the decoded original frame data.

[0087] The receiving and decoding process of the video stream is processed in ARM. First, the ZLMedia library is used to open and read the two video streams through the RTSP stream address of the RTSP stream address, and the encoder information of the two video streams is obtained. Then, two MPP decoders are registered according to the encoder information, where the first decoder is for the video stream of the first drone, and the second decoder is for the video stream of the second drone. The decoder operation mechanism is equivalent to the event registration mechanism. Specifically, when ZLMedia receives a complete H264 video stream frame, the H264 data decoding event responds. When a complete image frame is obtained by decoding, the post-decoding processing event responds. In the post-decoding event processing function, the decoded data is format converted and grayscale converted to obtain a grayscale image. In order to further improve the decoding efficiency, the decoded data is processed in a multi-threaded manner.

[0088] In this embodiment, ARM uses the PCIe interface to package the grayscale image data and transmit it to the FPGA, where the size of each data packet can be 16KB, 64KB, 128KB, etc. This embodiment uses 16KB, so a frame of image requires multiple data packets to be transmitted.

[0089] ARM's image data packaging process is as follows Figure 5 As shown, a complete grayscale image transmission includes a frame header for marking the start of the image, multiple small data packets. In this embodiment, each small data packet is 16KB, and the multiple small data packets are line-by-line scanned data of the original image data, that is, an image fragment. The frame tail is used to mark the end of the image transmission.

[0090] Step S3: FPGA extracts SIFT feature points from the cropped images R1 and R2 of the two flying devices, obtains SIFT feature point sets T1 and T2 of the two flying devices respectively, and transmits them to ARM;

[0091] The data processing flow between ARM and FPGA is as follows Figure 4As shown, data is transmitted between ARM and FPGA via PCIe;

[0092] ARM writes the image data to the BRAM of FPGA in a loop through PCIe. After FPGA finishes reading, it parses it to get the complete image data. After FPGA finishes SIFT feature point extraction, it writes the feature data to ARM through PCIe. After ARM finishes reading, it parses it to get the complete feature data.

[0093] The first flight device transmits the decoded frame data to the FPGA via PCIe. The grayscale image data size of a 1080P original video frame is 2MB. In order to improve the transmission efficiency, the data is transmitted in 16KB packets. ARM obtains data pixel by pixel from the grayscale image in a row scanning manner, generates a 16KB data packet, and writes the data into the BRAM of the FPGA via PCIe until the data transmission of an image frame is completed. The FPGA side fetches data from the BRAM of the FPGA via PCIe. After a complete grayscale image is read, the SIFT feature is calculated. After the calculation is completed, the feature data is packetized in 16KB size, and the data is written into the ARM via PCIe, and read by the ARM side.

[0094] The process of FPGA SIFT feature point extraction includes: constructing an image pyramid, and in this embodiment, a four-layer feature pyramid is specifically used; then calculating the direction of each feature point, and for the features of each direction of the feature point, calculating the corresponding feature descriptor; the feature stores the position, scale and direction of the feature point, and the feature descriptor stores the normalized feature vector. Figure 6 As shown, specifically, after FPGA reads the cropped grayscale image from BRAM, it first uses the Gaussian kernel function to perform Gaussian pyramid preprocessing on the cropped grayscale image, wherein the Gaussian kernel function is pre-stored in the FPGA as a constant; after processing, images of different scales are obtained and stored in the global memory; secondly, feature point detection and positioning are performed on images of different scales; then, by calculating the gradient histogram of the midpoint in the neighborhood of the feature point, the gradient amplitude and direction of the feature point are statistically analyzed, the main direction of the feature point is assigned, the direction of the point with the maximum gradient is selected as the main direction of the feature point, the feature point in the main direction is selected as the main feature point, and the position, scale, and direction features of the main feature point are recorded; next, the gradient of the neighborhood points of the main feature point is extracted to form a feature descriptor; finally, the position, scale, and direction features of the main feature point and the feature descriptors of the neighborhood points of the main feature point are packaged together as a SIFT feature vector to form a SIFT feature point set, which is output to the ARM.

[0095] Table 2 shows the efficiency comparison of the SIFT feature point extraction method based on FPGA of the present invention compared with the SIFT feature extraction operator provided by OpenCV and the extraction operator of VukanSIFT acceleration library. It can be seen from the table that the extraction acceleration method based on the present invention greatly reduces the extraction time. Since the SIFT feature point extraction based on FPGA of the present invention is real-time, combined with the calculation of the pose conversion matrix in the rough matching, the large error caused by the time delay between the pose matrix and the feature point alignment in the rough matching is effectively avoided, which provides support for the high-precision implementation of the present invention.

[0096]

[0097] After calculation and comparison, the image processing delay of a frame with a resolution of 720P is about 1364.32ms when SIFT feature point extraction is performed directly based on ARM. The image processing delay of a frame with a resolution of 720P is about 122.77ms when SIFT feature point extraction is performed based on the ARM Vulkan SIFT acceleration library. After FPGA acceleration processing, the image processing delay of a frame with a resolution of 720P is about 20.44ms.

[0098] In the specific implementation, the present invention does not implement all the feature extraction processes in FPGA. The steps of constructing the scale space of the image, generating the image Gaussian pyramid, establishing the feature point direction histogram, and calculating the main direction are accelerated by FPGA; while the steps of image color space conversion, normalization of descriptor feature vectors, etc. use CPU and RGA hardware module multi-core parallelization for higher efficiency.

[0099] Compared with other embedded processors such as ARM, the biggest advantage of FPGA image processing is the parallel pipeline processing method. The present invention proposes to divide the image into blocks and process them in parallel. For each block of the image, scale space is established, key points are extracted, key point feature descriptors are generated, and non-maximum value suppression is performed. Image segmentation is Block and The time required for each block is and , the formula is as follows:

[0100] in: N is the total number of pixels in each image, The delay of each process processing module in the chip, It is the highest frequency that each process processing module in the chip can achieve.

[0101] The traditional SIFT algorithm generates a 128-dimensional feature descriptor. The feature descriptor has too high a dimension, which results in redundant feature information, increases computational cost and complexity, and occupies useless resources. Therefore, the present invention optimizes the feature descriptor generation process to improve matching efficiency. First, a circle is drawn in a square area, and then the circular area is divided into 16 sub-domains, and the histogram is calculated separately. This histogram is the sum of the values ​​distributed in 8 directions for each pixel. The descriptor area is changed from a rectangle to a circle, which makes good use of the rotation invariance of the circular area, omits the calculation process of rotating the original rectangular area in the main direction, and can obtain the distribution of the gradient histogram after rotation by shifting the horizontal coordinate of the gradient histogram. At the same time, the original SIFT algorithm has too high a dimension and contains a lot of dimensional information, resulting in many wrong points being selected in the matching. Reducing the descriptor from 128 dimensions to 64 dimensions can reduce the corresponding matching time and reduce the amount of data processing.

[0102] Step S4: ARM maps the SIFT feature point set T1 of the first flying device to the second flying device based on the posture conversion matrix H of the two flying devices, performs rough matching, and obtains the mapping point set t1 mapped to the second flying device;

[0103] Step S5: Selecting a SIFT feature point set t2 in the SIFT feature point set T2 of the second flying device that is in the neighborhood of the mapping point set t1, and performing precise matching between the mapping point set t1 and the SIFT feature point set t2 to obtain a feature point matching pair;

[0104] For the feature point pairs of the first drone view Any feature point in , according to the image coordinates of the feature points , the pose transformation matrix calculates the image position corresponding to the second drone view , in the view of the second drone, with the coordinate point as the center, search for the most similar feature point in its 100×100 neighborhood , as the optimal matching point of the feature point of the first drone view, thereby obtaining a feature point matching pair of the first drone and the second drone view;

[0105] According to the above method, all feature points of the first drone view are searched to obtain all feature point matching pairs. .

[0106] Step S6: Calculate the homography matrix based on the feature point matching pairs, and send the homography matrix to the ground display and control platform;

[0107] The transformation matrix of the feature points is calculated based on the obtained feature point matching pairs as the homography matrix of the two-machine view puzzle.

[0108] Due to the weak texture background, the structural features of the two flight device views are not very obvious, and the extracted SIFT feature points are relatively similar. In order to further improve the matching accuracy, the present invention obtains a more accurate homography matrix of the two flight device views based on image feature fine matching on the basis of coarse matching.

[0109] The rough image matching based on pose information only involves processing the pose information of the flying device, so as to preliminarily obtain the transformation matrix of the two flying device views. Since there is no need to extract the feature information of the image, only the pose transformation of the image is performed. This method is very efficient and fast, and the processing time is generally in milliseconds. The rough pose matching method based on pose information has a good matching effect on the flat ground taken from above, but the matching effect is limited for two images at different depths, such as ground buildings and targets.

[0110] Step S7: The ground display and control platform performs large-field stitching of the views of the two flight devices according to the homography matrix.

[0111] After the ARM of the first UAV completes the matching of the feature points of the video streams of the two UAVs, it obtains the homography matrix and transmits the homography matrix to the ground display and control platform via UDP. The protocol package adopts the Json format, and the protocol content includes parameters such as timestamp and homography matrix.

[0112] The present invention is specifically aimed at multi-machine large-field-of-view video stitching under weak-textured background. It performs coarse image matching based on posture information, and performs precise image matching on feature point positions, gradient direction histograms, etc. according to similarity. Fine image matching is performed on the basis of coarse image matching. It can solve the problems of low accuracy and high real-time requirements for large-field-of-view video stitching under weak-textured background, and difficulty in extracting feature points under weak-textured background.

[0113] The relevant contents of the present invention were finally verified by actual flight of two-aircraft collaborative large-field stitching under typical weak-texture backgrounds such as jungle and water. Figure 7 shows the real-time stitching effect of the embodiment of the present invention under typical weak-texture backgrounds such as wasteland, water, and jungle during actual flight.

[0114] Those skilled in the art may make various modifications and variations to the present invention. If these modifications and variations are within the scope of the claims of the present invention and their equivalents, then these modifications and variations are also within the protection scope of the present invention.

[0115] The contents not described in detail in the specification are prior art known to those skilled in the art.

Claims

1. A multi-machine large field of view fast cascade stitching method with weak texture background, characterized by: The following steps are involved: Step S1: ARM acquires the posture data of the first flying device and the second flying device in real time, and obtains the posture conversion matrix H of the two flying devices according to the posture data; Step S2: ARM receives the video data of the first flying device and the second flying device in real time, and decodes them to obtain grayscale images of the two flying devices; solves the overlapping ROI area of ​​the grayscale images based on the posture conversion matrix H; crops the grayscale images of the two flying devices based on the overlapping ROI area to obtain cropped grayscale images R1 and R2 of the two flying devices; and transmits the cropped grayscale images R1 and R2 to FPGA; Step S3: FPGA extracts SIFT feature points from the cropped grayscale images R1 and R2 of the two flying devices, obtains SIFT feature point sets T1 and T2 of the two flying devices, and transmits them to ARM; Step S4: ARM maps the SIFT feature point set T1 of the first flying device to the second flying device based on the posture conversion matrix H of the two flying devices, performs rough matching, and obtains the mapping point set t1 mapped to the second flying device; Step S5: Selecting a SIFT feature point set t2 in the SIFT feature point set T2 of the second flying device that is in the neighborhood of the mapping point set t1, and performing precise matching between the mapping point set t1 and the SIFT feature point set t2 to obtain a feature point matching pair; Step S6: Calculate the homography matrix based on the feature point matching pairs, and send the homography matrix to the ground display and control platform; Step S7: the ground display and control platform performs large-field stitching of the views of the two flight devices according to the homography matrix; The system for multi-machine large field of view fast cascade stitching method applied to weak texture background includes a first flying device, a second flying device and a ground display and control platform. The first flying device is equipped with a data computing platform. The first flying device is wirelessly connected to the second flying device. The first flying device and the second flying device are respectively connected to the ground display and control platform. The data computing platform includes ARM and FPGA. ARM is connected to FPGA, and the first flight device and the second flight device are in master-slave following mode; the first flight device includes a first UAV, a first pod, a first airborne switch, a first flight control device, and a first airborne radio, and the first UAV is installed on the first pod; the data computing platform is connected to the first airborne switch, and the first airborne switch is respectively connected to the first pod, the first flight control device, and the first airborne radio; the second flight device includes a second UAV, a second pod, a second airborne switch, a second flight control device, and a second airborne radio, and the second UAV is installed on the second pod, and the second airborne switch is respectively connected to the second pod, the second flight control device, and the second airborne radio; the first airborne radio is wirelessly connected to the second airborne radio; the first airborne radio and the second airborne radio are respectively connected to the ground display and control platform; the first UAV and the second UAV are equipped with an inertial navigation unit and a GPS.

2. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In step S1, the method of obtaining the posture conversion matrix H of the two flying devices is as follows: First flight unit in position Take a picture of the ground scene and get an image , the second flying device is in position Take a picture of the ground scene and get an image ; For a point in the scene, Indicates that M is in position The camera coordinates, Indicates that M is in position The camera coordinates of For M in the image The corresponding point on For M in the image The corresponding points on the graph are as follows: , then the matrix H is called the image arrive The pose transformation matrix is is a non-zero constant term; according to the imaging principle of the camera, the pose conversion matrix between two aerial images can be expressed as: （1） Where K is the intrinsic parameter matrix of the camera, and the first drone is at position and the second drone in position The movement between , express The identity matrix of for The rotation matrix of is the three-dimensional translation vector; is the unit normal vector of the ground; is the distance from the coordinate origin to the geodetic plane; The location where the drone was taken It is expressed by 6 parameters, denoted as , They represent the deflection angle, pitch angle, and roll angle of the drone relative to the true north. Respectively represent the relative coordinate origin of the drone x , y , z Axis coordinates, i=1 or 2, where the origin of the coordinates is a point with known longitude and latitude, with due north as y Axis, due east x Axis, perpendicular to xy The plane is vertically upward z axis; according to , Sure and , the specific calculation rules are calculated according to Euler's formula: （3） （4） in: The Euler angles for yaw, pitch, and roll, respectively.

3. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In step S2, the specific method for calculating the overlapping ROI area is: S2-1: Extract and match the features of the entire starting frame to obtain the stitching result of the first frame and the overlapping areas of both frames; for the stitching of two images, is an interior point of image 1, is the interior point of image 2, and the set of interior point pairs is: ,in is the inlier pair of inlier 1 in image 1, is an interior point in image 1 The inner point pairs of the two images are sorted to obtain the points at the upper left corner and the lower right corner of the overlapping area of ​​the two images, and the overlapping area is obtained; ， in is the coordinate point of the upper left corner of the overlapping area, is the coordinate point of the lower right corner of the overlapping area; S2-2: In the next frame, feature extraction and matching are performed in the obtained overlapping area, and a new spliced ​​image and overlapping area are obtained for use in the next frame; S2-3: Iterate to obtain a stable overlapping region as the final overlapping RIO region.

4. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In step S2, a multi-threaded processing mechanism is used to realize the decoding of the video data of the two machines, the acquisition of image frames, and the calculation of overlapping ROI areas and image cropping based on the posture of the drones. The details are as follows: two threads are designed. Thread 1 obtains the postures of the two drones and the pod parameters to calculate the relative posture transformation matrix; Thread 2 obtains the video data for decoding; Thread 1 caches the calculated posture transformation matrix in a global variable in real time for Thread 2 to read.

5. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In step S2, the first pod and the second pod have the same viewing angle, which is set to 90° for aerial photography. The videos shot by the first pod and the second pod are in H264 stream format. ARM uses the RK3588 chip with an MPP hard decoding module, which can complete real-time decoding of the H264 format code stream through programming.

6. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In step S3, the specific process of FPGA performing SIFT feature point extraction includes: first, using the Gaussian kernel function to perform Gaussian pyramid preprocessing on the cropped grayscale image, and obtaining images of different scales after processing; second, detecting and locating the feature points of images of different scales; then, by calculating the gradient histogram of the midpoint in the neighborhood of the feature point, the gradient amplitude and direction of the feature point are statistically analyzed, the main direction of the feature point is assigned, the direction of the point with the maximum gradient is selected as the main direction of the feature point, the feature point in the main direction is selected as the main feature point, and the position, scale, and direction features of the main feature point are recorded; next, the gradient of the neighborhood points of the main feature point is extracted to form a feature descriptor; finally, the position, scale, and direction features of the main feature point and the feature descriptors of the neighborhood points of the main feature point are packaged together as a SIFT feature vector to form a SIFT feature point set.

7. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In steps S4 and S5, the specific methods of rough matching and fine matching are: For the feature point pairs of the first drone view Any feature point in , according to the image coordinates of the feature points , the pose transformation matrix calculates the image position corresponding to the second drone view , in the view of the second drone, with the coordinate point as the center, search for the most similar feature point in its 100×100 neighborhood , as the optimal matching point of the feature point of the first drone view, thereby obtaining a feature point matching pair of the first drone and the second drone view; all feature points of the first drone view are searched in the above manner to obtain all feature point matching pairs .

8. The multi-machine large field of view fast cascade stitching method for weak texture background as claimed in claim 1, characterized in that: In step S6, a transformation matrix of the feature points is calculated based on the obtained feature point matching pairs as the homography matrix of the two-machine view mosaic.

Images