Image fusion method, electronic device, unmanned aerial vehicle and storage medium

Abstract

Embodiments of the present application relate to the field of image processing, and particularly relate to an image fusion method, an electronic device, a UAV and a storage medium. The image fusion method comprises: obtaining a first image; obtaining a second image; obtaining a mapping image of the first image based on a first transformation parameter and the first image, the first transformation parameter being used to represent a spatial conversion relationship between the first image and the second image; and weighting and superimposing the second image and the mapping image to obtain a first fusion image. In the embodiments of the present application, the first image and the second image are obtained, and the first image is transformed based on the first transformation parameter, so that the mapping image of the obtained first image is aligned with the second image in terms of spatial structure, that is, the pixel points at the same positions in the mapping image and the second image have a corresponding relationship. In this way, the mapping image and the second image can be fused by directly weighting and adding, and the method is simple.

Description

[Technical Field]

[0001] This invention relates to the field of image processing, and more particularly to an image fusion method, electronic device, drone, and storage medium. [Background Technology]

[0002] Both low-light imaging and thermal imaging are good low-light imaging techniques. Thermal infrared imaging uses a radiometer to receive the infrared radiation emitted by the target and convert it into a visible thermal infrared image. Thermal infrared imaging is sensitive to temperature, can effectively detect targets with temperature differences, and has a long detection range, unaffected by environmental factors, lighting, or tree obstructions. However, it is not sensitive to changes in brightness, resulting in low-light image resolution and limited ability to depict details. It cannot identify or detect detailed features such as license plates or human faces.

[0003] Low-light imaging is a reflection imaging method, and its visual perception is close to that of visible light. It can effectively identify objects with different reflectivities to visible light, and low-light images are rich in detail and have a strong sense of depth. However, low-light imaging is greatly affected by factors such as weather, light source, and smoke, resulting in unstable imaging and possibly failing to work in rainy, foggy, or completely dark environments.

[0004] The fused image obtained by fusing low-light images and thermal infrared images combines the advantages of both. It can show the rich details of the low-light image and the infrared radiation image, which is different from the reflection image, thus greatly improving the observation capability of low-light video images. Current methods for fusing low-light images and thermal infrared images are relatively complex. [Summary of the Invention]

[0005] This application provides an image fusion method, an electronic device, a drone, and a storage medium. The image fusion method is relatively simple.

[0006] To address the aforementioned technical problems, embodiments of this application provide an image fusion method, comprising:

[0007] Get the first image;

[0008] Obtain the second image;

[0009] Based on the first transformation parameters and the first image, a mapped image of the first image is obtained, wherein the first transformation parameters are used to characterize the spatial transformation relationship between the first image and the second image;

[0010] The second image and the mapped image are weighted and superimposed to obtain a first fused image.

[0011] In some embodiments, the method further includes:

[0012] Extract contour feature points from the second image to obtain a contour image;

[0013] or,

[0014] Extract contour feature points from the mapped image to obtain a contour image.

[0015] In some embodiments, it also includes:

[0016] The first fused image and the contour image are weighted and superimposed to obtain a second fused image.

[0017] In some embodiments, it also includes:

[0018] In response to the first operation, the first image is used as a reference image, and when the first image is used as a reference image, the contour image includes contour feature points in the second image;

[0019] or,

[0020] In response to the second operation, the second image is used as a reference image, and when the second image is used as a reference image, the contour image includes contour feature points in the mapped image.

[0021] In some embodiments, it also includes:

[0022] In response to the third operation, the weights of the weighted superposition are adjusted.

[0023] In some embodiments, it also includes:

[0024] In response to the first image and the second image satisfying a first condition, the first image is used as a reference image;

[0025] In response to the first image and the second image satisfying the second condition, the second image is used as the reference image;

[0026] When the first image is used as a reference image, the contour image includes contour feature points in the second image; when the second image is used as a reference image, the contour image includes contour feature points in the mapped image.

[0027] The first condition includes one of the following conditions:

[0028] (1) The entropy value of the first image is greater than or equal to the entropy value of the second image;

[0029] (2) The sum of the pixel values ​​of the first image is greater than or equal to the sum of the pixel values ​​of the second image;

[0030] The second condition includes one of the following:

[0031] (3) The entropy value of the first image is less than the entropy value of the second image;

[0032] (4) The sum of the pixel values ​​of the first image is less than the sum of the pixel values ​​of the second image.

[0033] In some embodiments, it also includes:

[0034] The weights of the weighted superposition are determined based on whether the first image and the second image satisfy the first condition or the second condition.

[0035] The first condition includes one of the following conditions:

[0036] (1) The entropy value of the first image is greater than or equal to the entropy value of the second image;

[0037] (2) The sum of the pixel values ​​of the first image is greater than or equal to the sum of the pixel values ​​of the second image;

[0038] The second condition includes one of the following:

[0039] (3) The entropy value of the first image is less than the entropy value of the second image;

[0040] (4) The sum of the pixel values ​​of the first image is less than the sum of the pixel values ​​of the second image.

[0041] In some embodiments, the first image is a low-light image acquired by a low-light imaging device, and the second image is a thermal infrared image acquired by a thermal infrared imaging device.

[0042] or,

[0043] The first image is the thermal infrared image, and the second image is the low-light image.

[0044] In some embodiments, it also includes:

[0045] The brightness of the environment in which the low-light imaging device is located is obtained. If the brightness is greater than or equal to a first brightness threshold, the low-light image is used as a reference image; otherwise, the thermal infrared image is used as a reference image.

[0046] When the low-light image is used as a reference image, the contour image includes contour feature points in the thermal infrared image; when the thermal infrared image is used as a reference image, the contour image includes contour feature points in the mapped image of the low-light image.

[0047] In some embodiments, it also includes:

[0048] The weights of the weighted superposition are determined based on the brightness of the environment in which the low-light imaging device is located.

[0049] In some embodiments, the first transformation parameter includes a projection matrix, wherein the projection matrix is:

[0050]

[0051] Where m0, m1, m2, m3, m4, m5, m6, m7, m8, a, b, c, and d are constants, and L is the distance from the target to the imaging device.

[0052] Secondly, embodiments of this application also provide an electronic device, including:

[0053] A processor and a memory communicatively connected to the processor;

[0054] The memory stores computer program instructions, which, when invoked by the processor, cause the processor to execute the method described above.

[0055] Thirdly, embodiments of this application also provide a drone, including:

[0056] The fuselage is equipped with a first imaging device and a second imaging device, wherein the first imaging device is used to acquire a first image and the second imaging device is used to acquire a second image.

[0057] The arm is connected to the machine body;

[0058] A power unit, located on the arm, is used to provide the drone with the power for flight; and

[0059] Processor; memory communicatively connected to the processor;

[0060] The memory stores computer program instructions, which, when invoked by the processor, cause the processor to execute the method described above.

[0061] Fourthly, embodiments of this application also provide a storage medium storing computer-executable instructions for causing a processor to perform the method described above.

[0062] This application embodiment acquires a first image and a second image. The first image is transformed based on a first transformation parameter so that the mapped image of the acquired first image is spatially aligned with the second image; that is, pixels at the same position in the mapped image and the second image correspond to each other. Thus, the mapped image and the second image can be fused by direct weighted addition, a simple method. [Attached Image Description]

[0063] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0064] Figure 1 This is a schematic diagram illustrating the structure of an electronic device according to an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the structure of the drone according to an embodiment of this application;

[0066] Figure 3 This is a schematic diagram illustrating the inventive principle of an embodiment of this application;

[0067] Figures 4-5b This is a flowchart of the image fusion method according to an embodiment of this application;

[0068] Figures 6a-6b This is a schematic flowchart of the image fusion method according to an embodiment of this application;

[0069] Figure 7 This is a schematic diagram of image fusion in an embodiment of this application;

[0070] Figures 8a-10c This is a flowchart of the image fusion method according to an embodiment of this application;

[0071] Figure 11 This is a schematic diagram of the human-computer interaction interface in an embodiment of this application;

[0072] Figure 12 This is a schematic diagram of the human-computer interaction interface in an embodiment of this application.

Detailed Implementation Methods

[0073] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0074] Furthermore, the technical features involved in the various embodiments of this application described below do not conflict with each other and can be combined with each other.

[0075] Furthermore, although functional modules are divided in the device diagram and the logical order is shown in the flowchart, in some cases, the module division may differ from that in the device, or the execution order in the flowchart may differ from the steps shown or described. In addition, the terms "first," "second," and "third" used in this document do not limit the data or execution order; they are merely used to distinguish identical or similar items with essentially the same function and purpose.

[0076] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0077] Electronic devices often use multiple imaging devices to acquire images. These multiple imaging devices are based on different imaging technologies, which allows the images acquired by electronic devices to integrate the advantages of multiple imaging technologies and make up for each other's shortcomings.

[0078] Image fusion combines images from different imaging devices to obtain a more complete image or scene, improving the information content and clarity of the image by processing complementary information between multiple images.

[0079] Figure 1 The structure of an electronic device 100 is illustrated using two imaging devices as an example. The electronic device 100 includes a first imaging device 10 and a second imaging device 20. The first imaging device 10 is used to acquire a first image, and the second imaging device 20 is used to acquire a second image.

[0080] The electronic device also includes a processor 30 and a memory 40. The first imaging device 10 and the second imaging device 20 are both communicatively connected to the processor 30. The processor 30 and the memory 40 are connected via a line. Figure 1 In the embodiment shown, the first imaging device 10, the second imaging device 20, and the memory 30 are all connected to the processor 40 via a bus.

[0081] The memory 40 is used to store software programs, computer-executable program instructions, etc. The memory 40 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device, etc.

[0082] The memory 40 can be a read-only memory (ROM), or other types of static storage devices that can store static information and instructions, or random access memory (RAM), or other types of dynamic storage devices that can store information and instructions, or electrically erasable programmable read-only memory (EEPROM), and the specific type is not limited here.

[0083] For example, the aforementioned memory 40 can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM). This memory 40 can exist independently but is connected to the processor 30. Optionally, the memory 40 can also be integrated with the processor 30, for example, integrated within one or more chips.

[0084] In some embodiments, memory 40 may optionally include memory remotely located relative to processor 30, and this remote memory may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] The processor 30 connects various parts of the entire electronic device 100 using various interfaces and lines. By running or executing software programs stored in the memory 40 and calling data stored in the memory 40, it performs various functions of the electronic device and processes data, such as performing image processing on the first image and the second image, fusing the first image and the second image to obtain a fused image, etc. The processor 30 can implement the methods described in any embodiment of this application.

[0086] The processor 30 can be a field-programmable gate array (FPGA), digital signal processor (DSP), central processing unit (CPU), or graphics processing unit (GPU), or other functional units or modules with image processing capabilities.

[0087] Processor 30 can be a single-core processor or a multi-core processor. For example, processor 30 can be composed of multiple FPGAs or multiple DSPs. Furthermore, processor 30 can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Processor 30 can be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it can form a system-on-a-chip (SoC) with other circuits (such as codec circuits, hardware acceleration circuits, or various bus and interface circuits), or it can be integrated as a built-in processor within an application-specific integrated circuit (ASIC). This ASIC with integrated processor can be packaged separately or together with other circuits.

[0088] The first imaging device 10 and the second imaging device 20 can be imaging devices based on different imaging technologies, so that the electronic device can simultaneously possess the advantages of two imaging technologies. The first imaging device 10 and the second imaging device 20 can be various thermal imaging devices, optical imaging devices, etc., such as infrared thermal imagers, low-light night vision devices, charge-coupled device (CCD) image sensors, and complementary metal-oxide-semiconductor (CMOS) image sensors, etc.

[0089] Electronic equipment includes various instruments and devices composed of electronic components such as integrated circuits, transistors, and vacuum tubes, such as various drones, robots, camera equipment, and smart terminals.

[0090] Figure 2 This illustration shows one structure of a drone when the electronic device is a drone. The methods of the various embodiments of this application can also be applied to drones. Figure 2 As shown, the UAV 100 includes a fuselage 50, an arm 60 connected to the fuselage 50, a power unit 70 disposed on the arm 60, and a first imaging device 10 and a second imaging device 20 disposed on the fuselage 50.

[0091] In this embodiment, the first imaging device 10 and the second imaging device 20 are disposed on the fuselage 50. In other embodiments, they may also be disposed in other locations of the UAV, such as on the arm. Figure 2The diagram only schematically shows that the drone includes a first imaging device 10 and a second imaging device 20. In some applications, the first imaging device 10 and the second imaging device 20 can be set in multiple directions, such as in four directions, six directions, or eight directions.

[0092] The power unit 70 includes, for example, an electric motor and a propeller connected to the motor. The motor shaft rotates to drive the propeller to rotate, thereby providing lift to the drone.

[0093] In other embodiments, the drone 100 may also include a vision system (not shown) for acquiring images of the surrounding environment, identifying targets, detecting depth information of targets, and acquiring environmental maps, etc. The vision system may include a vision chip.

[0094] Drones may also include a flight controller (not shown in the figure). The flight controller is the control center of the drone and is used to coordinate and control the various components or elements of the drone to realize the various functions of the drone, such as flying, landing, taking pictures, and tracking targets.

[0095] In the case of an unmanned aerial vehicle (UAV), the processor 30 is the processor in the UAV, and the memory 40 is the internal memory in the UAV, or an external memory that is communicatively connected to the UAV. The methods of the various embodiments of this application can be executed by the flight controller in the UAV, by the vision chip in the vision system, by other controllers in the UAV, or by two or more controllers coordinating their execution to implement the methods of any embodiment of this application.

[0096] As will be understood by those skilled in the art, the above is merely an example of the hardware structure of the UAV 100. In practical applications, more components can be added to the UAV 100 according to actual functional needs. Of course, one or more components can also be omitted according to functional needs.

[0097] Among them, drones can be any suitable type of drone, such as fixed-wing drones, rotary-wing drones, unmanned airships, unmanned hot air balloons, etc.

[0098] The following description uses the first imaging device 10 and the second imaging device 20 as examples of a thermal infrared imaging device (e.g., an infrared thermal imager) and a low-light imaging device (e.g., a low-light night vision device).

[0099] Both low-light imaging and thermal imaging are good low-light imaging techniques. Thermal infrared imaging uses a radiometer to receive the infrared radiation emitted by the target and convert it into a visible thermal infrared image. Thermal infrared imaging is sensitive to temperature, can effectively detect targets with temperature differences, and has a long detection range, unaffected by environmental factors, lighting, or tree obstructions. However, it is not sensitive to changes in brightness, resulting in low resolution and limited ability to depict details. It cannot identify or detect detailed features such as license plates or human faces.

[0100] Low-light imaging is a reflection imaging method, and its visual perception is close to that of visible light. It can effectively identify objects with different reflectivities to visible light, and low-light images are rich in detail and have a strong sense of depth. However, low-light imaging is greatly affected by factors such as weather, light source, and smoke, resulting in unstable imaging and possibly failing to work in rainy, foggy, or completely dark environments.

[0101] The fused image obtained by fusing low-light images and thermal infrared images can combine the advantages of both. It can show the rich details of the low-light image and the infrared radiation image that is different from the reflected image, thus greatly improving the observation capability of low-light video images.

[0102] Current image fusion methods, such as those based on pyramid decomposition and wavelet analysis, are often quite complex. The image fusion method provided in the embodiments of this application obtains a mapped image of the first image using first transformation parameters and the first image. This mapped image is spatially aligned with the second image. Then, the second image and the mapped image are weighted and superimposed to obtain the fused image. This fusion method is relatively simple and can improve software running speed.

[0103] Please refer to Figure 3 Both the first and second images consist of M rows and N columns of pixels, each pixel having a pixel value. Pixel positions can be represented by pixel coordinates, which can be a sequentially arranged array of values ​​to characterize the position of each pixel within the image. For example... Figure 3 As shown, Figure 3 In the text, (0,0), (0,1), ..., (2,2) are all pixel coordinates.

[0104] A pixel value is a value assigned by a computer when an image is digitized, representing the average brightness information of a particular pixel in the image. The images involved in the embodiments of this application can be single-channel images, three-channel images, four-channel images, or images with more than one channel.

[0105] Single-channel images, also known as grayscale images, represent each pixel with a single pixel value. If the pixel value of a single-channel image is represented using 8 bits, the range of pixel values ​​is 0 (black) to 255 (white). Three-channel images generally refer to RGB images, which can represent both color and black-and-white images. If the pixel value of a three-channel image is represented using 8 bits, each pixel value is represented by three channel values: red (0-255), green (0-255), and blue (0-255) superimposed. Four-channel images add a luminance channel to the three-channel image to represent transparency.

[0106] As will be understood by those skilled in the art, if an image is a multi-channel image, then the pixel value of a pixel is composed of multiple channel values, and the same calculation needs to be performed on each channel value during the calculation process.

[0107] The aforementioned mapped image is spatially aligned with the second image, which can be understood as meaning that each pixel in the mapped image corresponds to each pixel in the second image. That is, pixels at the same location in both the mapped and second images point to the same feature point. Since the spatially aligned mapped and second images have corresponding pixels at the same location, their pixel values ​​can be directly weighted and superimposed to obtain the first fused image.

[0108] In this embodiment of the application, in order to achieve spatial alignment between the mapped image of the first image and the second image, the coordinates of each pixel in the first image are not directly changed. Instead, the pixel values ​​of each pixel in the first image are changed to achieve the effect of pixel position migration (e.g., coordinate change), thereby aligning the obtained mapped image with the second image in space.

[0109] In this embodiment of the application, the pixel value of each pixel is changed by a first transformation parameter, so as to... Figure 3 For example, after the first image I(x,y) is transformed by the first transformation parameter, a mapped image I0(x,y) corresponding to the second image V(x,y) is obtained. After the first transformation parameter, the pixel values ​​of each pixel point I(0,0), I(0,1), ..., I(2,2) of the first image become I0(0,0), I0(0,1), ..., I0(2,2). In the mapped image I0(x,y) and the second image V(x,y), I0(0,0) corresponds to V(0,0), I0(0,1) corresponds to V(0,1), ..., I0(2,2) corresponds to V(2,2).

[0110] The following describes the image fusion methods of various embodiments of this application. For ease of explanation, the following uses a drone as the electronic device, each image as a single-channel image, and the pixel value as a grayscale value as an example.

[0111] like Figure 4 As shown, the image fusion method provided in this application includes:

[0112] 101: Get the first image.

[0113] 102: Obtain the second image.

[0114] The first image can be a thermal infrared image acquired by a thermal infrared imaging device, and the second image can be a low-light image acquired by a low-light imaging device. In other embodiments, the first image can also be a low-light image acquired by a low-light imaging device, and the second image can also be a thermal infrared image acquired by a thermal infrared imaging device. Those skilled in the art will understand that the first image and the second image can also be any images based on different imaging technologies.

[0115] 103: Based on the first transformation parameters and the first image, obtain the mapped image of the first image.

[0116] After acquiring the first image, it is transformed based on the first transformation parameters so that the mapped image of the first image is spatially aligned with the second image, meaning that pixels at the same location in the mapped image and the second image correspond to each other. Thus, the mapped image and the second image can be fused by direct weighted addition, a simple method.

[0117] The first transformation parameter is used to characterize the spatial transformation relationship between the first image and the second image. After the first image is transformed using the first transformation parameter, the resulting mapped image is spatially aligned with the second image.

[0118] The first transformation parameter is, for example, the projection matrix. The projection matrix can be a matrix of any size. For example, the projection matrix is:

[0119]

[0120] Where m0, m1, m3, and m4 represent the scaling and rotation of the image, respectively; m2 represents the horizontal displacement of the image; m5 represents the vertical displacement of the image; m6 and m7 represent the deformation of the image in the horizontal and vertical directions, respectively; and m8 is a weighting factor, which is 1 under normalization conditions in some applications.

[0121] The parameters in the projection matrix can be obtained through calibration. Taking the first image as a thermal infrared image and the second image as a low-light image as an example, a calibration pattern with roughly the same texture information in thermal infrared imaging and low-light imaging can be selected to facilitate subsequent feature point matching.

[0122] The general process of obtaining the projection matrix is ​​as follows: the thermal infrared imaging device and the low-light imaging device take images at a certain distance (e.g., 6 meters, hereinafter referred to as the base distance) from the calibration pattern to obtain the first calibration image and the second calibration image. Feature points are extracted from the first and second calibration images respectively, and feature point matching is performed to obtain multiple matching feature point pairs. Then, based on the horizontal and vertical coordinate values ​​of each matching feature point pair, the parameter values ​​m0-m7 in the projection matrix are calculated.

[0123] Specifically, feature points such as edges and contours can be extracted from the first and second calibration images, and feature point matching can be performed to obtain each matched feature point pair. The Canny edge detection algorithm can be used to detect edges, contours, and other feature points in the image. In addition, other existing feature point matching methods can also be used for feature point matching.

[0124] In practical applications, the projection matrix can be calibrated before leaving the factory and added to the factory settings of each electronic device.

[0125] In other embodiments, to improve the matching accuracy of the mapped image and the second image and adapt to the distance transformation from the target to the UAV, the projection matrix may also be:

[0126]

[0127] Where L is the distance from the target to the imaging device, and a, b, c, and d are constants that can be obtained through calibration.

[0128] The projection matrix can be obtained based on the projection matrix H at the base distance, as well as L and parameters a, b, c, and d. Since the distances between the thermal infrared imaging device and the low-light imaging device are usually roughly the same, L can be either the distance from the target to the thermal infrared imaging device or the distance from the target to the low-light imaging device.

[0129] In practical applications, there are various methods to obtain the distance from the target to the imaging device, such as laser ranging or radar ranging.

[0130] With the first and second imaging devices fixed, the main manifestation of targets at different distances L in the image is the change in the horizontal and vertical displacements m2 and m5 of the image. Let m2 at distance L be m 2L Let m5 be the distance from L. 5L .

[0131] Parameters a, b, c, and d can be obtained by acquiring multiple calibration distances L and m. 2L The parameter pairs, and multiple calibration distances L and m 5LThe parameter pairs are obtained through linear fitting (e.g., least squares method). Where m 2L =am² + bL, which is m at a distance L. 5L =cm5+dL.

[0132] Multiple calibration distances are available, such as 1 meter, 2 meters, 3 meters...100 meters, etc. During each measurement, the distance between the calibration patterns of the thermal infrared imaging device and the low-light imaging device is adjusted.

[0133] For example: when the thermal infrared imaging device and the low-light imaging device are calibrated with a spacing of 1 meter, the first calibration image and the second calibration image are acquired, and m is obtained using the same method as described above for acquiring m0-m7. 21 m 51 To obtain the parameter pair (1, m) 21 ) and (1, m 51 ).

[0134] When the thermal infrared imaging device and the low-light imaging device are calibrated at a distance of 2 meters, the first calibration image and the second calibration image are acquired, and the parameter pair (2, m) is obtained. 22 ) and (2, m 52 ).

[0135] Measure again at other calibration distances (e.g., 3 meters, 4 meters, 5 meters... 100 meters) to obtain a greater number of parameter pairs.

[0136] Different electronic devices, namely different thermal infrared imaging devices and low-light imaging devices, can also be used to obtain more parameter pairs.

[0137] Then, parameters a, b, c, and d are obtained through linear fitting.

[0138] The following example, using projection matrix H, illustrates the mapping process of the first image.

[0139] In this embodiment, the grayscale value of each pixel in the first image is changed by the projection matrix to achieve the effect of pixel position migration, thereby aligning the obtained mapped image with the second image in space.

[0140] Assuming the first image is I(x, y) and the mapped image is I0(x, y), the gray value of any pixel (x, y) in the mapped image can be obtained as follows:

[0141] a: First, obtain the grayscale values ​​of the pixels at the same location in the first image and all surrounding pixels. In the case of a 3×3 projection matrix, the grayscale values ​​of a total of 9 pixels can be obtained. This forms a 3×3 matrix as follows:

[0142]

[0143] b: Multiplying I1 by the projection matrix H also yields a 3×3 matrix, as follows:

[0144]

[0145] c: I0(x, y) can be the average of the 9 elements in N1.

[0146] Taking I0(1,1) in the mapping image as an example, obtain I(0,0), I(0,1), I(0,2), I(1,0), I(1,1), I(1,2), I(2,0), I(2,1) and I(2,2) in the first image. Then I0(1,1)=(I(0,0)+I(0,1)+I(0,2)+I(1,0)+I(1,1)+I(1,2)+I(2,0)+I(2,1)+I(2,2)) / 9.

[0147] By calculating the gray value of any pixel in the mapped image using the gray values ​​of pixels at the same location in the first image and the gray values ​​of all surrounding pixels, a filtering effect can be achieved, removing the influence of noise.

[0148] In some embodiments, pixels at the edges of the mapped image, such as pixels with coordinates (0,0), (1,0), (2,0), (0,1), (0,2), etc., may not have their grayscale values ​​acquired, but may instead have their grayscale values ​​directly copied from those of their neighboring pixels.

[0149] By using the above method, the grayscale value of each pixel in the mapped image can be obtained, thus obtaining the mapped image.

[0150] As will be understood by those skilled in the art, the above is merely an example of the projection mapping process of the first image and should not be construed as a limitation on the projection mapping method. In other embodiments, other methods can be used to obtain the mapped image using the projection matrix.

[0151] 104: Weighted superposition of the second image and the mapped image to obtain the first fused image.

[0152] In some embodiments, the first fused image F(x, y) can be obtained by equation (1):

[0153]

[0154] Where w is the image width, h is the image height, a is the weight, and V(x, y) is the second image.

[0155] Let's take F(1,1) and F(1,2) as examples:

[0156] F(1,1)=I0(1,1)*a+V(1,1)*(1-a)

[0157] F(1,2)=I0(1,2)*a+V(1,2)*(1-a)

[0158] In other embodiments, to give the fused image more detail and features, after obtaining the first fused image, a contour image can also be obtained, and the first fused image and the contour image can be fused to obtain a second fused image. The second fused image, which incorporates the contour image, has richer features and details.

[0159] In this embodiment, the first image can be used as the reference image, and the weight of the first image can be set to be greater than the weight of the second image (or, depending on the specific application requirements, the weight of the first image may be set to be less than the weight of the second image). The contour image is taken from the second image. Please refer to this embodiment. Figure 5a The fusion method includes:

[0160] 101: Get the first image.

[0161] 102: Obtain the second image.

[0162] 103: Based on the first transformation parameters and the first image, obtain the mapped image of the first image.

[0163] 104: Weighted superposition of the second image and the mapped image to obtain the first fused image.

[0164] 105a: Extract contour feature points from the second image to obtain a contour image.

[0165] 106: Weighted superposition of the first fused image and the contour image to obtain a second fused image.

[0166] Figure 6a Taking the first image as a thermal infrared image and the second image as a low-light image as an example, the image fusion process when the first image is used as the reference image is shown.

[0167] In other embodiments, the second image can also be used as a reference image, and the weight of the second image can be set to be greater than the weight of the first image (depending on the specific application requirements, the weight of the first image may also be set to be greater than the weight of the second image). The contour image is taken from the mapped image of the first image. Please refer to this embodiment. Figure 5b The fusion method includes:

[0168] 101: Get the first image.

[0169] 102: Obtain the second image.

[0170] 103: Based on the first transformation parameters and the first image, obtain the mapped image of the first image.

[0171] 104: Weighted superposition of the second image and the mapped image to obtain the first fused image.

[0172] 105b: Extract contour feature points from the mapped image to obtain a contour image.

[0173] 106: Weighted superposition of the first fused image and the contour image to obtain a second fused image.

[0174] Figure 6b Taking the first image as a thermal infrared image and the second image as a low-light image as an example, the image fusion process is shown when the second image is used as the reference image.

[0175] Specifically, in some embodiments, feature points such as edges and contours in an image can be detected using algorithms such as the Canny edge detection algorithm. The extraction of contour feature points, as mentioned above, can be achieved by retaining the grayscale values ​​of the contour feature points while setting the grayscale values ​​of the remaining pixels to 0. In other embodiments, the detection and extraction of contour feature points can also be achieved using other algorithms.

[0176] In some embodiments, fusing the first fused image F(x, y) and the contour image V0(x, y) can be done with reference to the following formula:

[0177]

[0178] Where V1 is the grayscale threshold and E(x, y) represents the second fused image. If the grayscale value of a pixel in the contour image is greater than V1, it is considered a contour feature point and is fused with the first fused image. If the grayscale value of a pixel in the contour image is less than V1, it is not fused with the first fused image.

[0179] Taking E(1,1) and E(1,2) as examples, let V1 be 235. If V0(1,1) is greater than 235 and V0(1,2) is less than 235, then:

[0180] E(1,1)=F(1,1)*a+V0(1,1)*(1-a)

[0181] E(1,2)=F(1,2)

[0182] In situations where drones possess multiple imaging devices, the resulting fused image can combine the advantages of various imaging technologies. However, during image fusion, one image is often used as the reference image, while another one or more images serve as auxiliary images. The reference image typically retains more features in the fused image.

[0183] For example, Figure 6a In the embodiment shown, the thermal infrared image is used as the reference image, and the contour image is taken from the low-light image. The weight of the thermal infrared image can be set to be greater than that of the low-light image, so that the obtained fused image will retain more features of the thermal infrared image.

[0184] Please refer to Figure 7 , Figure 7 The first scenario shown (first row of images) uses a thermal infrared image as the reference image, with a greater weight than the low-light image, and the contour image is taken from the low-light image. The resulting fused image retains more features from the thermal infrared image. This first scenario is mainly used for UAV nighttime detection, as it can clearly mark the thermal contour information of the target, facilitating the identification of dynamic thermal targets.

[0185] Figure 7 The second scenario shown (third row of images) uses the low-light image as the reference image, with a greater weight than the thermal infrared image, and the contour image is taken from the thermal infrared image. The resulting fused image retains more features from the low-light image. This second scenario is mainly used for UAV reconnaissance and detection, where the image needs to depict the thermal contour of the target while providing more texture information in the detected target image.

[0186] Figure 7 The third scenario shown is (intermediate image), where the thermal infrared image and the low-light image have the same weight, both being 0.5.

[0187] In some embodiments of this application, the weight values ​​and / or the reference image can be changed in conjunction with environmental information of the UAV (e.g., brightness, weather conditions, etc.) and the required image requirements. The following explanation uses a thermal infrared image and a low-light image as examples of the first and second images.

[0188] The system can detect the brightness near the low-light imaging device. When the brightness is high, greater than or equal to a first brightness threshold, it indicates that the imaging conditions of the low-light imaging device are good, and the low-light image can be selected as the reference image, with the contour image taken from the thermal infrared image. When the brightness is less than the first brightness threshold, the thermal infrared image can be selected as the reference image, with the contour image taken from a mapped image of the low-light image. Please refer to this embodiment. Figure 8a and Figure 8b .

[0189] In some embodiments, when the brightness is greater than or equal to a first brightness threshold, the weight of the low-light image can be set to be greater than the weight of the thermal infrared image, for example, Figure 6a , Figure 6b In this context, 1-a > a. When the brightness is less than the first brightness threshold, the weight of the low-light image can be set to be less than the weight of the thermal infrared image. Please refer to... Figure 8c .

[0190] One of the features is a brightness detection device that can be placed near the low-light imaging device to detect the brightness in the vicinity of the low-light imaging device.

[0191] In other embodiments, a reference image and / or weights may be determined based on factors such as the image quality of the first and second images, for example, which image contains more features.

[0192] If the first image contains more features than the second image, then the first image is used as the reference image, and the contour image is taken from the second image. If the second image contains more features, then the second image is used as the reference image, and the contour image is taken from the mapped image of the first image.

[0193] In some embodiments, if the first image contains more features than the second image contains, then the weight value of the first image is greater than the weight value of the second image; conversely, if the second image contains more features, then the weight value of the second image is greater than the weight value of the first image. For example, in the above embodiments, a > 1 - a.

[0194] Specifically, in some embodiments, the presence of more features in an image can be determined by factors such as its entropy value and the sum of its pixel values. Taking entropy as an example, if the entropy value of the first image is greater than or equal to the entropy value of the second image, then the first image is used as the reference image, and / or the weight of the first image is set to be greater than the weight of the second image. If the entropy value of the first image is less than the entropy value of the second image, then the second image is used as the reference image, and / or the weight of the second image is set to be greater than the weight of the first image. Please refer to this embodiment. Figures 9a-9c .

[0195] Taking the sum of pixel values ​​as an example, if the sum of pixel values ​​in the first image is greater than or equal to the sum of pixel values ​​in the second image, then the first image is used as the reference image, and / or the weight of the first image is set to be greater than the weight of the second image (e.g., a > 1 - a). If the sum of pixel values ​​in the first image is less than the sum of pixel values ​​in the second image, then the second image is used as the reference image, and / or the weight of the second image is set to be greater than the weight of the first image (e.g., 1 - a > a).

[0196] Image entropy calculation is a prior art technique and will not be elaborated upon here. The sum of image pixel values, for example, the sum of the pixel values ​​of all pixels in the image.

[0197] In some embodiments of this application, the user can select a reference image and / or set the weights of a first image and a second image through the human-machine interface provided on the drone. These settings can be made according to the actual application requirements.

[0198] Specifically, the user can perform a first operation through the human-computer interaction interface, selecting a first image as the reference image; or, the user can perform a second operation through the human-computer interaction interface, selecting a second image as the reference image. Alternatively, the user can perform a third operation through the human-computer interaction interface, adjusting the weights of the first or second image.

[0199] When the user performs the first operation, the drone responds by using the first image as a reference image. Please refer to the following embodiment for details. Figure 10a When the user performs the second operation, the drone responds by using the second image as a reference image. Please refer to the following embodiment for details. Figure 10b When the user performs a third operation, the drone responds by adjusting the weight of the first or second image. Please refer to the example provided. Figure 10c .

[0200] It should be noted that the weights of images related to the first image can be the same as those of the first image. For example, the projection image and contour image of the first image can be given the same weights as the first image. In cases where the first image accounts for a large proportion of the first fused image, the weights of the first fused image can also be the same as those of the first image.

[0201] The weights of images related to the second image can be the same as those of the second image. For example, the projection image and contour image of the second image can be assigned the same weights as the second image. In cases where the second image accounts for a larger proportion of the first fused image, the weights of the first fused image can also be the same as those of the second image.

[0202] The first, second, and third operations can be any operations that enable the drone to receive signals. For example, the first operation could be... Figure 11 The tap operation shown is the "first image selection box" click operation on the touch screen, and the second operation is, for example... Figure 11 The example shown is a single click on the "selection box for the second image" on the touchscreen. A third operation could be dragging a progress bar.

[0203] In practical applications, the third operation can be used to set the weight 'a' of the first image, and the weight of the second image is obtained by calculating 1-a. For example, when the user drags the progress bar and stops it at a certain point, the weight 'a' of the first image is set.

[0204] Since drones typically acquire images while in motion, the images acquired at intervals can vary significantly. To better fuse the first and second images, it's necessary to maintain consistency between the acquired images. In some embodiments, the acquisition times of the first and second images are made approximately the same.

[0205] The system can first acquire the timestamps of the first and second images. If their timestamps are not synchronized—for example, if the timestamp of the first image is later than the timestamp of the second image by a first time T—then the first image can be acquired after a delay of a first time T, allowing the drone to acquire both images synchronously. Similarly, if the timestamp of the second image is later than the timestamp of the first image by a first time T, the second image can be acquired after a delay of a first time T, ensuring the drone can acquire both images synchronously.

[0206] It should be noted that the numbers for each step above are only for identifying that step and do not indicate the order in which the steps are performed. The steps can be in any other order besides the order in which they are presented in the text.

[0207] This application also provides a computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 1 One of the processors 30 can enable the one or more processors to execute the image fusion method in any of the above method embodiments.

[0208] This application also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by a machine, cause the machine to perform the image fusion method of any of the above embodiments.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image fusion method, characterized by, The method comprises: acquiring a first image; acquiring a second image; acquiring a mapping image of the first image based on a first transformation parameter and the first image, the first transformation parameter being used to represent a spatial conversion relationship between the first image and the second image; weighting and superimposing the second image and the mapping image to acquire a first fusion image; in response to the first image and the second image satisfying a first condition, taking the first image as a reference image; in response to the first image and the second image satisfying a second condition, taking the second image as a reference image; extracting contour feature points in the second image to acquire a contour image; or extracting contour feature points in the mapping image to acquire a contour image; when the first image is taken as a reference image, the contour image comprises contour feature points in the second image, and when the second image is taken as a reference image, the contour image comprises contour feature points in the mapping image of the first image; wherein the first condition comprises one of the following conditions: (1) an entropy value of the first image is greater than or equal to an entropy value of the second image; (2) a sum of pixel values of the first image is greater than or equal to a sum of pixel values of the second image; the second condition comprises one of the following conditions: (3) the entropy value of the first image is less than the entropy value of the second image; (4) the sum of pixel values of the first image is less than the sum of pixel values of the second image.

2. The method of claim 1, wherein, Further comprising: weighting and superimposing the first fusion image and the contour image to acquire a second fusion image.

3. The method according to claim 1 or 2, characterized in that, Further comprising: in response to a first operation, taking the first image as a reference image, and when the first image is taken as a reference image, the contour image comprises contour feature points in the second image; or in response to a second operation, taking the second image as a reference image, and when the second image is taken as a reference image, the contour image comprises contour feature points in the mapping image.

4. The method according to claim 1 or 2, characterized in that, Further comprising: in response to a third operation, adjusting the weight of the weighting and superimposition.

5. The method according to claim 1 or 2, characterized in that, Further comprising: based on the first image and the second image satisfying the first condition or the second condition, determining the weight of the weighting and superimposition.

6. The method of claim 2, wherein, The first image is a micro-light image acquired by a micro-light imaging device, and the second image is a thermal infrared image acquired by a thermal infrared imaging device; or the first image is the thermal infrared image, and the second image is the micro-light image.

7. The method of claim 6, wherein, Further comprising: acquiring the brightness of the environment in which the micro-light imaging device is located, if the brightness is greater than or equal to a first brightness threshold, taking the micro-light image as a reference image, otherwise, taking the thermal infrared image as a reference image; when the micro-light image is taken as a reference image, the contour image comprises contour feature points in the thermal infrared image, and when the thermal infrared image is taken as a reference image, the contour image comprises contour feature points in the mapping image of the micro-light image.

8. The method of claim 7, wherein, Further comprising: based on the brightness of the environment in which the micro-light imaging device is located, determining the weight of the weighting and superimposition.

9. The method of claim 1 or 2, wherein, The first transformation parameter comprises a projection matrix, and the projection matrix is: wherein, , , , , , , , , , , b, c, d are constants, and L is the distance from the target to the imaging device.

10. An electronic device, comprising: Further comprising: a processor and a memory connected in communication with the processor. The memory stores computer program instructions which, when invoked by the processor, cause the processor to perform the method of any one of claims 1-9.

11. A drone, characterized in that, Comprise: A machine body, a first imaging device and a second imaging device are arranged on the machine body, the first imaging device is used for acquiring a first image, and the second imaging device is used for acquiring a second image; A machine arm connected with the machine body; A power device arranged on the machine arm, used for providing power for flight of the unmanned aerial vehicle; And A processor, a memory in communication connection with the processor; The memory stores computer program instructions which, when invoked by the processor, cause the processor to perform the method of any one of claims 1-9.

12. A storage medium, characterized by The storage medium stores computer executable instructions for causing the processor to perform the method of any one of claims 1-9.

Images