Search and rescue method based on airborne synthetic aperture imaging and white de-display technology

Abstract

The embodiment of the invention relates to the technical field of emergency rescue, in particular to a search and rescue method based on airborne synthetic aperture imaging and a white de-display technology, and the method comprises the steps: indicating an unmanned plane to carry out multi-position multi-view shooting above a current designated position; based on the shot thermal image, judging whether the current specified position is a suspected target position, and if yes, performing motion estimation on the shot RGB image; preprocessing the thermal image and the RGB image to obtain the thermal image and the RGB image with the same resolution, and carrying out white de-display processing; inputting the RGB image after white display removal and the corresponding thermal image into a multi-modal image fusion model to obtain a fused image; performing projection texture mapping and three-dimensional rendering by using an image rendering technology to obtain a synthetic image synthesized by perspective projection rendering corresponding to each fused image; adjusting each fused image to obtain an unshielded image; and determining whether the current suspected target position is included in the search and rescue route based on the de-occlusion image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of emergency rescue technology, and in particular to a search and rescue method based on airborne synthetic aperture imaging and white removal technology. Background Art

[0002] Recently, Ailao Mountain in Chuxiong, Yunnan Province, has become an instant hit on the Internet due to the adventure videos of Internet celebrity bloggers. The mysterious primeval forest, thrilling adventure elements, and strange scenes with great visual impact have all stimulated the nerves of travel and adventure enthusiasts. Many tourists and bloggers have followed suit and turned the little-known human restricted area into a bustling and noisy Internet celebrity scenic spot, and outdoor adventure has become a matter of going anywhere. However, hidden behind this is a shocking and painful lesson. According to incomplete statistics, in 2023 alone, there were 335 outdoor adventure accidents in China, 320 people were injured, 156 people died, and 26 people were missing. Such irresponsible outdoor adventures not only ignore their own lives, but also waste a lot of emergency rescue resources, and even bring irreparable damage to nature reserves.

[0003] How to rescue people in distress in mountainous areas and primeval forests has always been a difficult problem in emergency rescue, because it requires professional rescue teams to spend a lot of manpower and material resources, and the rescue success rate is not high. Even if many places begin to try paid rescue services, the fees collected cannot cover the costs incurred during the entire rescue process.

[0004] Drones are widely used in various emergency rescue and search operations such as earthquakes, fires, explosion prevention, and power line repairs due to their advantages of zero casualties, low cost, easy control, and good hovering performance. However, in remote mountain rescue and virgin forest rescue, the victims are covered by tall trees, which seriously blocks the drone's line of sight. The images taken by drones cannot penetrate the cover of tall trees and are therefore almost unusable.

[0005] To address this problem, domestic and foreign research teams have proposed some methods for de-occlusion imaging in order to capture the target object under the occlusion. For example, Dai et al. proposed a de-occlusion algorithm based on two modalities: event image and RGB image. First, an adaptive refocusing algorithm is used to refocus the RGB image, and the event data is also refocused to disperse the foreground occlusion. Then, the event data is segmented based on the RGB image timestamp and represented as event frames. The two modal data are input into a generation network composed of a two-branch feature extraction module, a feature fusion module, and an image reconstruction module. The image de-occlusion task is well implemented, and a clear and complete target image can be reconstructed.

[0006] However, event cameras are limited by spatial resolution and color images, and therefore perform poorly in scenarios such as mountain rescue and virgin forest rescue, which have large areas, many obstructions, and small targets. Even if RGB images are fused, it is difficult to find the location of the person in distress in the vast forest, so they are not very helpful for emergency rescue. Summary of the invention

[0007] In view of this, the embodiments of the present application propose a search and rescue method based on airborne synthetic aperture imaging and white removal technology, which can be used to carry out rescue assessment in mountainous areas, virgin forests and other places, determine the activities of persons in distress, clarify rescue targets, reduce the workload of the rescue team, and improve the success rate of emergency rescue.

[0008] In a first aspect, an embodiment of the present application proposes a search and rescue method based on airborne synthetic aperture imaging and white removal technology, comprising the following steps:

[0009] S1, instructs the UAV to arrive at the designated location and take multi-position and multi-view shooting of thermal images and RGB images over the current designated location;

[0010] S2, based on each thermal image obtained by shooting, determining whether the current designated position is a suspected target position, if the current designated position is determined to be a suspected target position, executing S3, otherwise, directly executing S10;

[0011] S3, using an incremental structure-from-motion algorithm to perform motion estimation on the captured RGB images, and obtaining rotation parameters and translation parameters of the camera corresponding to each RGB image;

[0012] S4, preprocessing the thermal image and RGB image including pairing registration and cropping operations to obtain thermal images and RGB images with the same resolution, and performing white removal processing on all RGB images in HSV space;

[0013] S5, inputting the RGB image after white removal and the corresponding thermal image into a pre-trained multimodal image fusion model to obtain a fused image;

[0014] S6, using image rendering technology, performing projection texture mapping and three-dimensional rendering according to the customized three-dimensional plane model, each fused image, and a view transformation matrix corresponding to each fused image, to obtain a composite image synthesized by perspective projection rendering corresponding to each fused image;

[0015] S7, adjusting the perspective projection corresponding to each fused image to achieve focusing at the current suspected target position in the synthesized image and blurring of the occluder, thereby obtaining a de-occluded image;

[0016] S8. Determine the confidence level of the current suspected target position based on the de-occluded image, and determine whether the confidence level of the current suspected target position is greater than a preset confidence threshold. If so, execute S9; otherwise, directly execute S10.

[0017] S9. Incorporate the current suspected target position into the search and rescue route.

[0018] S10. Instruct the drone to fly to the next designated position and perform multi-position and multi-view shooting of thermal images and RGB images above the next designated position.

[0019] Optionally, after the drone arrives at the designated position, it starts from the starting point and flies along a preset route until it reaches the return point and ends. The spacing range of the preset route is set from 1m to 2m, the height range is set from 20m to 30m, the shooting interval of the drone is set from 1m to 2m, and the heading angle of the drone is fixed after entering the starting point.

[0020] Optionally, based on the captured thermal images, determine whether the current designated position is a suspected target position, including:

[0021] Traverse each thermal image and traverse each pixel point in the current thermal image.

[0022] Determine whether the heat value of the current pixel point is greater than a first preset threshold, and incorporate the pixel points with heat values greater than the first preset threshold into the same connectivity domain to be determined.

[0023] Successively determine whether the contours of the connectivity domains to be determined in each thermal image are approximately human-shaped. If the number of thermal images with contours of the connectivity domains to be determined approximately human-shaped is greater than a second preset threshold, determine that the current designated position is a suspected target position; otherwise, determine that the current designated position is not a suspected target position.

[0024] Optionally, perform preprocessing on the thermal images and RGB images, including pairing registration and cropping operations, to obtain thermal images and RGB images with the same resolution, including:

[0025] For paired thermal images and RGB images, use OpenCV for image registration to obtain thermal images and RGB images with the same resolution.

[0026] Perform white de-revealing processing on all RGB images in the HSV space, including:

[0027] Convert the registered RGB images from the RGB color space to the HSV color space.

[0028] Based on a preset white threshold range, identify the white areas in the registered RGB images.

[0029] Reduce the brightness and saturation of the white area to make it appear gray or darker, and convert it back to the RGB color space to obtain the RGB image after white area disappearance.

[0030] Optionally, input the RGB image after white area disappearance and the corresponding thermal image into a pre-trained multi-modal image fusion model to obtain a fused image, including:

[0031] Input the RGB image after white area disappearance and the corresponding thermal image into an alpha fusion model, a pyramid fusion model or a Poisson fusion model to obtain a fused image.

[0032] Optionally, use image rendering technology to perform projective texture mapping and 3D rendering according to a custom 3D plane model, each fused image, and the view transformation matrix corresponding to each fused image to obtain a composite image synthesized by the perspective projection rendering corresponding to each fused image, including:

[0033] Respectively generate a number of virtual cameras based on the rotation parameters and translation parameters of the cameras corresponding to each fused image, traverse each virtual camera, and calculate the model view matrix corresponding to the current virtual camera based on the view transformation matrix corresponding to the current virtual camera and the model matrix corresponding to the custom 3D plane model;

[0034] Use the model view matrix corresponding to the current virtual camera to move the custom 3D plane model in front of the current virtual camera, set the field of view angle of the frustum using the resolution and focal length of the fused image corresponding to the current virtual camera, and then use a preset perspective projection matrix to generate the frustum corresponding to the current virtual camera, and place the fused image corresponding to the current virtual camera at the focal length of the frustum corresponding to the current virtual camera;

[0035] Select one of the virtual cameras as the rendering camera, use the other virtual cameras as projectors for rendering, the frustum corresponding to the camera as the rendering frustum, the frustums corresponding to each projector as projection frustums, set the custom 3D plane model at the focal length of the rendering camera, and project the perspective projections of each fused image on the custom 3D plane model inside the rendering frustum and each projection frustum;

[0036] Render the custom 3D plane model inside the rendering frustum to obtain a composite image synthesized by the perspective projection renderings corresponding to each fusion.

[0037] Optionally, adjust the perspective projections corresponding to each fused image to achieve focusing at the current suspected target position in the composite image and blurring of occluders to obtain an unoccluded image, including:

[0038] By adjusting the parameters in the view transformation matrix, the translation and rotation of the rendering camera and each projector are realized. The adjustable parameters in the view transformation matrix include position coordinates, direction vectors, and up vectors;

[0039] By adjusting the parameters in the perspective projection matrix, the change of the rendering range is realized. The adjustable parameters in the perspective projection matrix include the field of view angle and the clip planes, and the clip planes include the near clip plane and the far clip plane;

[0040] By adjusting the parameters in the model matrix, the translation and rotation of the custom three-dimensional plane model are realized;

[0041] Among them, during the process of adjusting the parameters in the view transformation matrix, the perspective projection matrix, and the model matrix, when the target at the specified position in the synthesized image is focused and the occluder is blurred, the adjustment is stopped to obtain the target-occlusion-removed image.

[0042] The present application proposes a search and rescue method based on airborne synthetic aperture imaging and white removal technology. The drone is used to shoot the designated location. Based on the thermal image, it is preliminarily determined whether there are suspected traces of activities of the person in distress at the designated location. If the designated location is determined to be a suspected target location, the synthetic aperture is simulated to achieve the de-occlusion of the suspected target location. Data is collected according to the drone image shooting method, and posture data and image data are obtained through a series of processing. The image is synthesized by an OpenGL-based image rendering method. The synthesized result is de-occluded and enhanced at the designated location by using an interactive interface, so as to determine the confidence of the suspected target location. If the confidence is high, it means that the location is likely to be the location of the person in distress, and the location needs to be included in the search and rescue route. Otherwise, go to the next designated location for shooting. The present application applies de-occlusion imaging to the field of emergency rescue. By shooting multi-modal image data and fusing the multi-modal image data for de-occlusion imaging, rescue research and judgment can be carried out in mountainous areas, virgin forests and other places, the location of the person in distress is determined, the rescue target is clarified, the workload of the rescue team is reduced, and the success rate of emergency rescue is improved. This application adopts a white removal and color segmentation method based on HSV. For scenes with rich colors or complex backgrounds, using the HSV color space for processing has natural advantages, and the operation is simple and fast. The white removal method based on HSV can darken the nearly white or pure white occluders in the RGB image, effectively preventing them from affecting the subsequent color segmentation. The color segmentation method based on HSV can effectively segment the high-brightness, white target information in the fused image, which is convenient for subsequent synthetic aperture imaging. This application designs three judgments. The first judgment is to determine whether the current position is the intended target position, the second judgment is to determine whether the suspected target position is included in the search and rescue route, and the third judgment is completed by the rescue team through on-site search, which well guarantees the life safety of the person in distress and also provides strong protection for the life safety of the rescue team.

[0043] In a second aspect, an embodiment of the present application proposes a search and rescue system based on airborne synthetic aperture imaging and white removal technology, the system comprising:

[0044] The navigation module is used to instruct the drone to arrive at the designated location and take multi-position and multi-view shooting of thermal images and RGB images over the current designated location;

[0045] An acquisition module is used to acquire thermal images and RGB images taken by a drone;

[0046] A primary determination module, used to determine whether the current designated position is a suspected target position based on each thermal image, and if the current designated position is determined to be a suspected target position, the motion estimation module is started, otherwise, the navigation module is started;

[0047] A motion estimation module, which is used to perform motion estimation on the captured RGB images by using an incremental structure from motion algorithm, and obtain the rotation parameters and translation parameters of the camera corresponding to each RGB image;

[0048] An image preprocessing module, which is used to preprocess the thermal images and RGB images, including operations such as paired registration and cropping, to obtain thermal images and RGB images with the same resolution;

[0049] A white de - highlighting module, which is used to perform white de - highlighting processing on all RGB images in the HSV space;

[0050] A fusion module, which is used to input the white - de - highlighted RGB images and the corresponding thermal images into a pre - trained multi - modal image fusion model to obtain fused images;

[0051] A synthesis module, which is used to utilize image rendering technology, and perform projective texture mapping and 3D rendering according to a custom 3D plane model, each fused image, and the view transformation matrix corresponding to each fused image, to obtain a synthesized image synthesized by perspective projection rendering corresponding to each fused image;

[0052] A defocusing and occlusion - removing module, which is used to adjust the perspective projection corresponding to each fused image to achieve focusing at the current suspected target position in the synthesized image and blurring of the occluder, to obtain an occlusion - removed image;

[0053] A secondary determination module, which is used to determine the confidence level of the current suspected target position based on the occlusion - removed image, and determine whether the confidence level of the current suspected target position is greater than a preset confidence level threshold. If so, the route planning module is started; otherwise, the navigation module is restarted;

[0054] A route planning module, which is used to incorporate the current suspected target position into the search and rescue route;

[0055] A navigation module, which is also used to instruct the UAV to fly to the next specified position and perform multi - position and multi - perspective shooting of thermal images and RGB images above the next specified position.

[0056] In a third aspect, an embodiment of the present application proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a search and rescue method based on airborne synthetic aperture imaging and white de - highlighting technology as described in the first aspect above.

[0057] Fourthly, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor, can implement a search and rescue method based on airborne synthetic aperture imaging and white de-revealing technology as described in the first aspect above.

[0058] It can be understood that the beneficial effects of the above second aspect to the fourth aspect can refer to the relevant descriptions in the first aspect above, and will not be elaborated here. Description of the Drawings

[0059] To more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present application or the related technology. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0060] Figure 1 It is a flowchart of a search and rescue method based on airborne synthetic aperture imaging and white de-revealing technology provided in an embodiment of the present application;

[0061] Figure 2 It is a schematic diagram of the unmanned aerial vehicle shooting process provided in an embodiment of the present application;

[0062] Figure 3 It is a schematic diagram of the principle of data initialization provided in an embodiment of the present application;

[0063] Figure 4 It is a schematic diagram of the principle of projection mapping texture provided in an embodiment of the present application;

[0064] Figure 5 It is a schematic diagram of the principle of three-dimensional rendering synthesis provided in an embodiment of the present application;

[0065] Figure 6 It is a schematic diagram of the effect of adjusting the view transformation matrix provided in an embodiment of the present application;

[0066] Figure 7 It is a schematic diagram of the effect of adjusting the perspective projection matrix provided in an embodiment of the present application;

[0067] Figure 8 It is a schematic diagram of the effect of adjusting the model matrix provided in an embodiment of the present application;

[0068] Figure 9 It is some de-occluded images provided in an embodiment of the present application;

[0069] Figure 10It is a schematic structural diagram of a search and rescue system based on airborne synthetic aperture imaging and white de - display technology provided in another embodiment of the present application;

[0070] Figure 11 It is a schematic structural diagram of an electronic device provided in another embodiment of the present application. Specific embodiments

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In various embodiments of the present application, many technical details are proposed for readers to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented. The following division of each embodiment is only for convenient description and should not constitute any limitation to the specific implementation manner of the present application. Each embodiment can be combined and cross - referenced with each other on the premise of no contradiction.

[0072] To solve the problem that the traditional method of de - occluding imaging is of little help to emergency rescue, an embodiment of the present application proposes a search and rescue method based on airborne synthetic aperture imaging and white de - display technology, which is applied to a server. The implementation details of a search and rescue method based on airborne synthetic aperture imaging and white de - display technology proposed in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution.

[0073] The specific process of a search and rescue method based on airborne synthetic aperture imaging and white de - display technology proposed in this embodiment can be as Figure 1 shown and includes:

[0074] S1, instruct the unmanned aerial vehicle (UAV) to reach the designated position and perform multi - position and multi - perspective shooting of thermal images and RGB images above the current designated position.

[0075] In specific implementation, when a distress event occurs, the rescue team will first plan several designated positions according to the previous activity routes of the person in distress. The person in distress may be at these designated positions. The purpose of this embodiment is to screen these designated positions, incorporate the designated positions with a high probability of the presence of the person in distress into the search and rescue route, and the rescue team will carry out emergency rescue according to the search and rescue route. This screening process is achieved by the UAV. The server instructs the UAV to reach the designated position and perform multi - position and multi - perspective shooting of thermal images and RGB images above the current designated position, so as to obtain a number of pairs of thermal images and RGB images, forming a thermal image set and an RGB image set.

[0076] In an example, the shooting process of the UAV is as Figure 2As shown, after the UAV arrives at the designated position, it will start from the starting point and fly according to the preset route until it reaches the return point and then ends and returns. The spacing range of the preset route is set from 1m to 2m, the height range is set from 20m to 30m, the shooting interval of the UAV is set from 1m to 2m, and the heading angle of the UAV is fixed after entering the starting point.

[0077] S2. Based on each thermal image obtained by shooting, determine whether the current designated position is a suspected target position. If it is determined that the current designated position is a suspected target position, then execute S3; otherwise, directly execute S10.

[0078] In a specific implementation, after the server obtains each thermal image and each RGB image obtained by the UAV shooting, it needs to determine whether the current designated position is a suspected target position based on each thermal image obtained by shooting. If it is determined that the current designated position is a suspected target position, then continue with the subsequent processing of the thermal image and the RGB image. If it is determined that the current designated position is not a suspected target position, then directly instruct the UAV to fly to the next designated position and perform multi-position and multi-view shooting of the thermal image and the RGB image above the next designated position.

[0079] In an example, when the server makes the first determination, it needs to traverse each thermal image and each pixel point in the current thermal image. Subsequently, determine whether the heat value of the current pixel point is greater than the first preset threshold, and incorporate the pixel points with heat values greater than the first preset threshold into the same connected domain to be determined. Next, sequentially determine whether the contours of the connected domains to be determined in each thermal image are approximately human-shaped. If the number of thermal images with contours of the connected domains to be determined approximately human-shaped is greater than the second preset threshold, then determine that the current designated position is a suspected target position; otherwise, determine that the current designated position is not a suspected target position. Among them, both the first preset threshold and the second preset threshold can be set by those skilled in the art according to actual needs, and this embodiment does not make specific limitations on this.

[0080] In an example, the determination of whether the contour of the connected domain is approximately human-shaped is achieved based on three aspects: morphological features, proportional relationships, and key structure points. The morphological features consist of geometric shapes, the number of limbs, neck transitions, joint markings, and concave and convex contours. The proportional relationships include the head-to-body ratio, limb lengths, and symmetry. The determination of key structure points can be achieved by comparing the similarity with a standard human key point model (such as the 17-point model of OpenPose).

[0081] S3. For the RGB images obtained by shooting, use the incremental structure from motion algorithm for motion estimation to obtain the rotation parameters and translation parameters of the camera corresponding to each RGB image.

[0082] In a specific implementation, after the server determines that the current specified position is a suspected target position, it can perform motion estimation on the captured RGB images using the incremental structure from motion algorithm to obtain the rotation parameters and translation parameters of the camera corresponding to each RGB image.

[0083] In one example, the server performs pose estimation on all the captured RGB image data using COLMAP to obtain a pose data text file, which contains the rotation parameters and translation parameters of the camera corresponding to each RGB image, denoted as QW, QX, QY, QZ, TX, TY, TZ.

[0084] S4. Preprocess the thermal image and the RGB image, including operations such as paired registration and cropping, to obtain thermal images and RGB images with the same resolution, and perform white de - highlighting processing on all RGB images in the HSV color space.

[0085] In a specific implementation, after the server completes the motion estimation of the RGB images, it is necessary to preprocess the thermal image and the RGB image, including operations such as paired registration and cropping, to obtain thermal images and RGB images with the same resolution, and perform white de - highlighting processing on all RGB images in the HSV color space.

[0086] In one example, the server uses OpenCV to perform image registration on all pairs of thermal images and RGB images to obtain thermal images and RGB images with the same resolution.

[0087] In one example, when the server performs white de - highlighting processing on all RGB images in the HSV color space, it first needs to convert the registered RGB images from the RGB color space to the HSV color space. Subsequently, based on a preset white threshold range, it identifies the white regions in the registered RGB images. Finally, it reduces the brightness and saturation of the white regions in the registered RGB images to make them appear gray or darker, and then converts them back to the RGB color space to obtain the RGB images after white de - highlighting. Among them, the preset white threshold range can be set by those skilled in the art according to actual needs.

[0088] S5. Input the RGB images after white de - highlighting and the corresponding thermal images into a pre - trained multi - modal image fusion model to obtain a fused image.

[0089] In a specific implementation, after the server completes the white de - highlighting operation, it is necessary to input the RGB images after white de - highlighting and the corresponding thermal images into a pre - trained multi - modal image fusion model to obtain a fused image.

[0090] In one example, the pre-trained multi-modal image fusion model can be selected from an alpha fusion model, a pyramid fusion model, a Poisson fusion model, etc.

[0091] S6. Using image rendering technology, based on the custom three-dimensional plane model, each fused image, and the view transformation matrix corresponding to each fused image, perform projective texture mapping and 3D rendering to obtain a composite image synthesized by the perspective projection rendering corresponding to each fused image.

[0092] In a specific implementation, after the server obtains the fused images, it needs to use image rendering technology to perform projective texture mapping and 3D rendering based on the custom three-dimensional plane model, each fused image, and the view transformation matrix corresponding to each fused image, to obtain a composite image synthesized by the perspective projection rendering corresponding to each fused image.

[0093] In one example, the server respectively generates a number of virtual cameras based on the rotation parameters and translation parameters of the cameras corresponding to each fused image, traverses each virtual camera, and calculates the model view matrix corresponding to the current virtual camera based on the view transformation matrix corresponding to the current virtual camera and the model matrix corresponding to the custom three-dimensional plane model. Next, using the model view matrix corresponding to the current virtual camera, move the custom three-dimensional plane model in front of the current virtual camera, set the field of view angle of the frustum using the resolution and focal length of the fused image corresponding to the current virtual camera, then use the preset perspective projection matrix to generate the frustum corresponding to the current virtual camera, and place the fused image corresponding to the current virtual camera at the focal length of the frustum corresponding to the current virtual camera. Subsequently, the server selects one of the virtual cameras as the rendering camera at will, uses the other virtual cameras as projectors for rendering, the frustum corresponding to the camera as the rendering frustum, the frustums corresponding to each projector as projection frustums, sets the custom three-dimensional plane model at the focal length of the rendering camera, and projects the perspective projections of each fused image onto the custom three-dimensional plane model inside the rendering frustum and each projection frustum. Finally, the server renders the custom three-dimensional plane model inside the rendering frustum to obtain a composite image synthesized by the perspective projection rendering corresponding to each fusion.

[0094] In one example, the server first performs "data initialization", and the principle of "data initialization" can be as Figure 3As shown. Based on the rotation parameters and translation parameters of the cameras corresponding to each fused image, the server generates a number of virtual cameras, traverses each virtual camera, and based on the view transformation matrix V corresponding to the current virtual camera and the model matrix D corresponding to the custom three-dimensional plane model, calculates the model view matrix G corresponding to the current virtual camera, where G = D × V. The server uses the model view matrix G corresponding to the current virtual camera to move the custom three-dimensional plane model in front of the current virtual camera, sets the field of view of the frustum using the resolution and focal length of the fused image corresponding to the current virtual camera, and then uses the preset perspective projection matrix P to generate the frustum corresponding to the current virtual camera, and places the fused image corresponding to the current virtual camera at the focal length of the frustum corresponding to the current virtual camera.

[0095] In one example, after "data initialization" is completed, the server can perform "projective mapping texture". The principle of "projective mapping texture" is as Figure 4 shown. The server arbitrarily selects one of the virtual cameras as the rendering camera, and the other virtual cameras serve as projectors. The frustum corresponding to the rendering camera is used as the rendering frustum, and the frustums corresponding to each projector are used as projection frustums. The server sets the custom three-dimensional plane model at the focal length of the rendering camera, and projects each fused image onto the custom three-dimensional plane model perspectively inside the rendering frustum and each projection frustum.

[0096] In one example, after "projective mapping texture" is completed, the server can perform "3D rendering synthesis". The principle of "3D rendering synthesis" is as Figure 5 shown. The server only renders the part of the custom three-dimensional plane model that is inside the rendering frustum, and does not render the part that is outside the rendering frustum. After rendering, the server obtains a composite image synthesized by the perspective projections corresponding to each fused image. Such a perspective projection rendering synthesis process can scientifically and accurately retain the information in each fused image that is useful for occluded imaging, thereby improving the quality of occluded imaging.

[0097] S7. Adjust the perspective projections corresponding to each fused image to achieve focusing at the current suspected target position in the composite image and blurring of the occluder, obtaining an occluded image.

[0098] In a specific implementation, after the server obtains the composite image, it can adjust the perspective projections corresponding to each fused image, thereby achieving focusing at the current suspected target position in the composite image and blurring of the occluder, obtaining an occluded image. This adjustment is performed dynamically. After the target at the specified position in the composite image achieves focusing and the occluder is blurred, the adjustment can be stopped to obtain an occluded image of the target.

[0099] In one example, the server realizes the translation and rotation of the rendering camera and each projector by adjusting the parameters in the view transformation matrix. The adjustable parameters in the view transformation matrix include position coordinates, direction vectors, and up vectors. By adjusting the parameters in the perspective projection matrix, the change of the rendering range is realized. The adjustable parameters in the perspective projection matrix include the field of view angle and the clipping plane, and the clipping plane includes the near clipping plane and the far clipping plane. By adjusting the parameters in the model matrix, the translation and rotation of the custom three-dimensional plane model are realized. Among them, during the process of adjusting the parameters in the view transformation matrix, the perspective projection matrix, and the model matrix, when the target at the specified position in the composite image is focused and the occluder is blurred, the adjustment is stopped to obtain the target-occlusion-removed image.

[0100] In one example, the effect of adjusting the view transformation matrix is as Figure 6 shown, and the effect of adjusting the perspective projection matrix is as Figure 7 shown, and the effect of adjusting the model matrix is as Figure 8 shown. The occlusion-removed image obtained by the server is as Figure 9 shown.

[0101] S8. Determine the confidence level of the current suspected target position based on the occlusion-removed image, and determine whether the confidence level of the current suspected target position is greater than the preset confidence level threshold. If so, execute S9; otherwise, directly execute S10.

[0102] S9. Incorporate the current suspected target position into the search and rescue route.

[0103] S10. Instruct the UAV to fly to the next specified position and perform multi-position and multi-view shooting of thermal images and RGB images above the next specified position.

[0104] In specific implementation, after the server obtains the occlusion-removed image, a second determination can be made, that is, determine the confidence level of the current suspected target position based on the occlusion-removed image, and determine whether the confidence level of the current suspected target position is greater than the preset confidence level threshold. If it is determined that the confidence level of the current suspected target position is greater than the preset confidence level threshold, the current suspected target position can be incorporated into the search and rescue route. If it is determined that the confidence level of the current suspected target position is less than or equal to the preset confidence level threshold, instruct the UAV to fly to the next specified position and perform multi-position and multi-view shooting of thermal images and RGB images above the next specified position.

[0105] In one example, the determination of the confidence level of the suspected target position is also the determination of whether the contour is approximately humanoid, which is also realized based on the proportional relationship and key structure points in three aspects.

[0106] The present embodiment proposes a search and rescue method based on airborne synthetic aperture imaging and white removal technology, which uses a drone to shoot a designated location, and preliminarily determines whether there are suspected traces of activities of a person in distress at the designated location based on the thermal image. If the designated location is determined to be a suspected target location, the synthetic aperture is simulated to achieve de-occlusion of the suspected target location. Data is collected according to the drone image shooting method, and posture data and image data are obtained through a series of processing. The image is synthesized using an OpenGL-based image rendering method, and the synthesized result is de-occluded and enhanced at the designated location by using an interactive interface, so as to determine the confidence of the suspected target location. If the confidence is high, it means that the location is likely to be the location of the person in distress, and the location needs to be included in the search and rescue route. Otherwise, go to the next designated location for shooting. This embodiment successfully applies de-occlusion imaging to the field of emergency rescue. By shooting multi-modal image data and fusing the multi-modal image data to perform de-occlusion imaging, it can be well used to carry out rescue assessment in mountainous areas, virgin forests and other places, determine the places where the victims have been active, clarify the rescue targets, reduce the workload of the rescue team, and improve the success rate of emergency rescue. This embodiment introduces HSV-based white removal and color segmentation in the process of de-occlusion imaging. For scenes with rich colors or complex backgrounds, using the HSV color space for processing has natural advantages, and the operation is simple and fast. The HSV-based white removal method can darken the nearly white or pure white occluders in the RGB image, effectively preventing them from affecting the subsequent color segmentation. The HSV-based color segmentation method can effectively segment the high-brightness, white target information in the fused image, which is convenient for subsequent synthetic aperture imaging. This embodiment is designed with three judgments. The first judgment is to determine whether the current position is the intended target position, the second judgment is to determine whether the suspected target position is included in the search and rescue route, and the third judgment is completed by the rescue team's on-site search, which well ensures the life safety of the person in distress and also provides strong protection for the life safety of the rescue team.

[0107] The step division of the above methods is only for clear description. When implemented, they can be combined into one step, or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0108] Another embodiment of the present application proposes a search and rescue system based on airborne synthetic aperture imaging and white removal technology. The implementation details of the search and rescue system based on airborne synthetic aperture imaging and white removal technology proposed in this embodiment are described in detail below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this example.

[0109] Figure 10 It is a schematic structural diagram of a search and rescue system based on airborne synthetic aperture imaging and white de - rendering technology proposed in this embodiment. The system includes: a navigation module M1, an acquisition module M2, a primary determination module M3, a motion estimation module M4, an image pre - processing module M5, a white de - rendering module M6, a fusion module M7, a synthesis module M8, a virtualization and occlusion removal module M9, a secondary determination module M10, and a route planning module M11.

[0110] The navigation module M1 is used to instruct the unmanned aerial vehicle M0 to reach a specified position and perform multi - position and multi - perspective shooting of thermal images and RGB images over the current specified position.

[0111] The acquisition module M2 is used to acquire each thermal image and each RGB image captured by the unmanned aerial vehicle M0.

[0112] The primary determination module M3 is used to judge whether the current specified position is a suspected target position based on each thermal image. If it is determined that the current specified position is a suspected target position, the motion estimation module M4 is started; if it is determined that the current specified position is not a suspected target position, the navigation module M1 is started.

[0113] The motion estimation module M4 is used to perform motion estimation on the captured RGB images using the incremental structure from motion algorithm to obtain the rotation parameters and translation parameters of the camera corresponding to each RGB image.

[0114] The image pre - processing module M5 is used to perform pre - processing on the thermal images and RGB images, including pairing registration and cropping operations, to obtain thermal images and RGB images with the same resolution.

[0115] The white de - rendering module M6 is used to perform white de - rendering processing on all RGB images in the HSV space.

[0116] The fusion module M7 is used to input the white - de - rendered RGB images and the corresponding thermal images into a pre - trained multi - modal image fusion model to obtain fused images.

[0117] The synthesis module M8 is used to perform projective texture mapping and 3D rendering using image rendering technology according to a custom 3D plane model, each fused image, and the view transformation matrix corresponding to each fused image, to obtain a synthetic image synthesized by perspective projection rendering corresponding to each fused image.

[0118] The virtualization and occlusion removal module M9 is used to adjust the perspective projection corresponding to each fused image to achieve focusing on the current suspected target position in the synthetic image and blurring of the occluder, to obtain an occlusion - removed image.

[0119] The secondary determination module M10 is configured to determine the confidence level of the current suspected target position based on the de-occluded image, and determine whether the confidence level of the current suspected target position is greater than a preset confidence threshold. If so, the route planning module M11 is activated; otherwise, the navigation module M1 is restarted.

[0120] The route planning module M11 is configured to incorporate the current suspected target position into the search and rescue route.

[0121] The navigation module M1 is further configured to instruct the drone M0 to fly to the next designated position and perform multi-position and multi-view shooting of thermal images and RGB images above the next designated position.

[0122] It is worth mentioning that each module involved in this embodiment is a logic module. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovative part of this application, units not closely related to solving the technical problems proposed in this application are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0123] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in cooperation with the above method embodiment. The relevant technical details and technical effects mentioned in the above method embodiment are still valid in this embodiment. To avoid repetition, they are not elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiment.

[0124] Another embodiment of the present application proposes an electronic device, the specific structure of which is as Figure 11 shown, including: at least one processor C1; and a memory C2 communicatively connected to the at least one processor C1; wherein, the memory C2 stores instructions executable by the at least one processor C1, and the instructions are executed by the at least one processor C1 so that the at least one processor C1 can execute a search and rescue method based on airborne synthetic aperture imaging and white de-exposure technology as described in the above method embodiment.

[0125] Among them, the memory and the processor can be connected in a bus manner. The bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and the memory together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described herein. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna. Further, the antenna also receives data and transmits the data to the processor.

[0126] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store the data used by the processor when executing operations.

[0127] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program, which when executed by a processor, can implement a search and rescue method based on airborne synthetic aperture imaging and white de-revealing technology as described in the above method embodiment.

[0128] That is, those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium, including several instructions for causing a device (such as a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disks, or optical discs and other various media that can store program codes.

[0129] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in practical applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A search and rescue method based on airborne synthetic aperture imaging and white de - display technology, characterized in that, Including: S1, instruct the drone to reach the designated position and perform multi-position and multi-view shooting of thermal images and RGB images above the current designated position; S2, based on the captured thermal images, determine whether the current designated position is a suspected target position. If it is determined that the current designated position is a suspected target position, execute S3; otherwise, directly execute S10; S3, for the captured RGB images, use the incremental structure from motion algorithm to perform motion estimation to obtain the rotation parameters and translation parameters of the camera corresponding to each RGB image; S4, perform preprocessing on the thermal images and RGB images, including paired registration and cropping operations, to obtain thermal images and RGB images with the same resolution, and perform white de-revealing processing on all RGB images in the HSV color space; S5, input the white de-revealed RGB images and the corresponding thermal images into a pre-trained multi-modal image fusion model to obtain fused images; S6, use image rendering technology to perform projective texture mapping and 3D rendering according to a custom 3D plane model, each fused image, and the view transformation matrix corresponding to each fused image to obtain a composite image synthesized by the perspective projection rendering corresponding to each fused image; S7, adjust the perspective projection corresponding to each fused image to achieve focusing on the current suspected target position in the composite image and blurring of the occluder to obtain an unoccluded image; S8, determine the confidence level of the current suspected target position based on the unoccluded image, and judge whether the confidence level of the current suspected target position is greater than a preset confidence level threshold. If so, execute S9; otherwise, directly execute S10; S9, include the current suspected target position in the search and rescue route; S10, instruct the drone to fly to the next designated position and perform multi-position and multi-view shooting of thermal images and RGB images above the next designated position.

2. The search and rescue method based on airborne synthetic aperture imaging and white de - rendering technology according to claim 1, characterized in that, After the drone reaches the designated position, it starts from the starting point and flies along a preset route until it reaches the return point and ends. The spacing range of the preset route is set from 1m to 2m, the height range is set from 20m to 30m, the shooting interval of the drone is set from 1m to 2m, and the heading angle of the drone is fixed after entering the starting point.

3. A search and rescue method based on airborne synthetic aperture imaging and white de - display technology as claimed in claim 1, characterized in that, Based on the captured thermal images, determining whether the current designated position is a suspected target position includes: Traverse each thermal image and traverse each pixel point in the current thermal image; Judge whether the heat value of the current pixel point is greater than a first preset threshold, and include the pixel points with heat values greater than the first preset threshold in the same to-be-determined connected domain; Successively judge whether the contours of the to-be-determined connected domains in each thermal image are approximately human-shaped. If the number of thermal images with to-be-determined connected domains whose contours are approximately human-shaped is greater than a second preset threshold, determine that the current designated position is a suspected target position; otherwise, determine that the current designated position is not a suspected target position.

4. The search and rescue method based on airborne synthetic aperture imaging and white de - rendering technology according to claim 1, characterized in that, Performing preprocessing on the thermal images and RGB images, including paired registration and cropping operations, to obtain thermal images and RGB images with the same resolution, including: For paired thermal images and RGB images, use OpenCV for image registration to obtain thermal images and RGB images with the same resolution; Perform white de - rendering processing on all RGB images in the HSV space, including: Convert the registered RGB image from the RGB color space to the HSV color space; Based on a preset white threshold range, identify the white regions in the registered RGB image; Reduce the brightness and saturation of the white regions to make them appear gray or darker, and then convert back to the RGB color space to obtain the RGB image after white de - rendering.

5. The search and rescue method based on airborne synthetic aperture imaging and white de - display technology according to claim 4, wherein, Input the RGB image after white de - rendering and the corresponding thermal image into a pre - trained multi - modal image fusion model to obtain a fused image, including: Input the RGB image after white de - rendering and the corresponding thermal image into an alpha fusion model, a pyramid fusion model, or a Poisson fusion model to obtain a fused image.

6. The search and rescue method based on airborne synthetic aperture imaging and white de - rendering technology according to claim 5, wherein, Using image rendering technology, perform projective texture mapping and 3D rendering according to a custom 3D plane model, each fused image, and the view transformation matrix corresponding to each fused image, to obtain a composite image synthesized by the perspective projection renderings corresponding to each fused image, including: Based on the rotation parameters and translation parameters of the camera corresponding to each fused image respectively, generate a number of virtual cameras, traverse each virtual camera, and calculate the model - view matrix corresponding to the current virtual camera based on the view transformation matrix corresponding to the current virtual camera and the model matrix corresponding to the custom 3D plane model; Using the model - view matrix corresponding to the current virtual camera, move the custom 3D plane model in front of the current virtual camera, set the field of view of the frustum using the resolution and focal length of the fused image corresponding to the current virtual camera, then use a preset perspective projection matrix to generate the frustum corresponding to the current virtual camera, and place the fused image corresponding to the current virtual camera at the focal length of the frustum corresponding to the current virtual camera; Select one of the virtual cameras as the rendering camera, use the other virtual cameras as projectors for rendering, the frustum corresponding to the camera as the rendering frustum, the frustums corresponding to each projector as projection frustums, set the custom 3D plane model at the focal length of the rendering camera, and project the perspective projections of each fused image onto the custom 3D plane model inside the rendering frustum and each projection frustum; Render the custom 3D plane model inside the rendering frustum to obtain a composite image synthesized by the perspective projection renderings corresponding to each fusion.

7. The search and rescue method based on airborne synthetic aperture imaging and white de - display technology according to claim 6, characterized in that, Adjust the perspective projections corresponding to each fused image to achieve focusing at the current suspected target position in the composite image and blurring of the occluder to obtain an unoccluded image, including: By adjusting the parameters in the view transformation matrix, achieve the translation and rotation of the rendering camera and each projector. The adjustable parameters in the view transformation matrix include position coordinates, direction vectors, and up vectors; By adjusting the parameters in the perspective projection matrix, achieve a change in the rendering range. The adjustable parameters in the perspective projection matrix include the field of view and the clip planes. The clip planes include the near clip plane and the far clip plane; By adjusting the parameters in the model matrix, achieve the translation and rotation of the custom 3D plane model; Among them, in the process of adjusting the parameters in the view transformation matrix, the perspective projection matrix, and the model matrix, when the target at the specified position in the synthesized image is focused and the occluder is blurred, stop the adjustment to obtain a target-occlusion-removed image.

8. A search and rescue system based on airborne synthetic aperture imaging and white de - display technology, characterized in that, It includes: A navigation module, configured to instruct the drone to reach the specified position and perform multi-position and multi-view shooting of thermal images and RGB images above the current specified position; An acquisition module, configured to acquire each thermal image and each RGB image captured by the drone; A primary determination module, configured to determine whether the current specified position is a suspected target position based on each thermal image. If it is determined that the current specified position is a suspected target position, start the motion estimation module; otherwise, start the navigation module; A motion estimation module, configured to perform motion estimation on the captured RGB images using the incremental structure from motion algorithm to obtain the rotation parameters and translation parameters of the camera corresponding to each RGB image; An image preprocessing module, configured to perform preprocessing including paired registration and cropping operations on the thermal images and RGB images to obtain thermal images and RGB images with the same resolution; A white dehighlighting module, configured to perform white dehighlighting processing on all RGB images in the HSV space; A fusion module, configured to input the white dehighlighted RGB images and the corresponding thermal images into a pre-trained multi-modal image fusion model to obtain fused images; A synthesis module, configured to use image rendering technology to perform projective texture mapping and 3D rendering according to a custom 3D plane model, each fused image, and the view transformation matrix corresponding to each fused image to obtain a synthesized image synthesized by perspective projection rendering corresponding to each fused image; A blur and occlusion removal module, configured to adjust the perspective projection corresponding to each fused image to achieve focusing on the current suspected target position in the synthesized image and blurring of the occluder to obtain an occlusion-removed image; A secondary determination module, configured to determine the confidence level of the current suspected target position based on the occlusion-removed image and determine whether the confidence level of the current suspected target position is greater than a preset confidence level threshold. If so, start the route planning module; otherwise, restart the navigation module; A route planning module, configured to incorporate the current suspected target position into the search and rescue route; The navigation module is further configured to instruct the drone to fly to the next specified position and perform multi-position and multi-view shooting of thermal images and RGB images above the next specified position.

9. An electronic device, characterized in that, It includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, instructions executable by the at least one processor are stored in the memory, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute a search and rescue method based on airborne synthetic aperture imaging and white dehighlighting technology as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a search and rescue method based on airborne synthetic aperture imaging and white dehighlighting technology as described in any one of claims 1 to 7.

Images