Unmanned aerial vehicle four-light camera module picture fusion method and system

Abstract

The application relates to a UAV four-light camera module picture fusion method, which comprises the following steps: calibrating an infrared module, an ultraviolet module, a visible light module and a laser module, acquiring internal and external parameters of each module, respectively performing distortion correction and focal length normalization processing on original images of each module based on the internal parameters, mapping each module image into a visible light module coordinate system in combination with external parameters and distance information of a fault point measured in real time by the laser module, taking a picture frame of the visible light module as a reference to calculate an overlapping area of the registered images, and performing boundary padding on a non-overlapping area to obtain a target image group, calculating a dynamic fusion weight based on a pixel gradient entropy and a correlation coefficient of each module and the visible light module, and performing weighted fusion on the target image group according to the dynamic fusion weight to obtain a fusion image. Through the application, the problem of four-light fusion picture deviation of a UAV is solved, high-precision registration of four-light images is realized, and there is no picture deviation.

Description

Technical Field

[0001] This application relates to the field of image fusion, and in particular to a method and system for image fusion of a UAV four-light camera module. Background Technology

[0002] The use of drones equipped with a four-light camera module (infrared + ultraviolet + visible light + laser) has become the mainstream configuration for inspecting power grid line faults. It can simultaneously capture the thermal radiation (infrared), corona discharge (ultraviolet), appearance details (visible light), and distance information (laser) of the fault point.

[0003] However, due to the center spacing of the optical lenses in the four-light module (usually 5-15mm), the difference in focal length (8-12mm for infrared, 10-15mm for ultraviolet, and 6-10mm for visible light), and the deviation in installation posture, the pixel offset of the same fault point in the images of different modules reaches 8-15px, and the image boundary misalignment exceeds 20px, which seriously affects the multi-dimensional feature fusion and accurate positioning of the fault point.

[0004] Existing four-light module image fusion typically focuses only on single-dimensional correction. For example, it uses fixed pixel translation registration, without considering the impact of focal length changes on image scaling, resulting in image shift. Furthermore, the boundary processing is simplified, and fusion is achieved by cropping overlapping areas, leading to a loss of effective field of view. In addition, existing algorithms are mostly designed for two-light / three-light modules, ignoring the auxiliary role of laser distance information in multi-light registration, resulting in insufficient fusion accuracy. Summary of the Invention

[0005] This application provides a method, system, electronic device, and storage medium for fusing images from a UAV four-light camera module, to at least solve the problem of image offset in the fusing of UAV four-light camera modules in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for image fusion using a four-light camera module of a drone, the method comprising: The infrared module, ultraviolet module, visible light module and laser module are calibrated to obtain the intrinsic and extrinsic parameters of each module. Based on the intrinsic parameters, the original images of each module are subjected to distortion correction and focal length normalization to obtain the corrected infrared image, ultraviolet image, visible light image and laser image. Using the visible light module as a reference, and combining the external parameters and the fault point distance information measured in real time by the laser module, the corrected infrared image, ultraviolet image and laser image are mapped to the visible light module coordinate system to obtain the registered images of each module; Using the image frame of the visible light module as a reference, the overlapping area of ​​the registered images of each module is calculated, and the non-overlapping area is filled in by an interpolation method based on distance attenuation weight to obtain the target image group. Based on pixel gradient entropy and the correlation coefficient between each module and the visible light module, a dynamic fusion weight is calculated for each pixel in the image of each module, and the target image group is weighted and fused according to the dynamic fusion weight to obtain a fused image.

[0007] In some embodiments, the intrinsic parameters include the effective focal length of each module, the principal point coordinates of the image, the radial distortion coefficient, and the tangential distortion coefficient; the extrinsic parameters include the rotation matrix and translation vector of each module relative to the visible light module.

[0008] In some embodiments, the distortion correction and focal length normalization processing of the original images of each module based on the intrinsic parameters includes: Convert the pixel coordinates in the original image pixel coordinate system to normalized coordinates in the normalized image coordinate system; The normalized coordinates are corrected for distortion based on the radial distortion coefficient and the tangential distortion coefficient to obtain distortion-free normalized coordinates; The distortion-free normalized coordinates are back-projected onto the pixel coordinate system to obtain the distortion-free pixel coordinates; Based on the effective focal length of the visible light module, the ratio of the effective focal length of each module to the effective focal length of the visible light module is calculated, and the distortion-free pixel coordinates of each module are multiplied by this ratio to normalize the focal length of all module images.

[0009] In some embodiments, mapping the corrected infrared image, ultraviolet image, and laser image to the visible light module coordinate system to obtain the registered images of each module includes: The pixel coordinates of the corrected infrared, ultraviolet and laser images are transformed into the camera coordinate system of the corresponding module through the inverse matrix of the intrinsic parameter matrix of the module, so as to obtain the three-dimensional point coordinates in the camera coordinate system of each module. The three-dimensional point coordinates are rotated and transformed according to the rotation matrix to an intermediate coordinate system aligned with the orientation of the visible light module; Based on the translation vector and the fault point distance measured in real time by the laser module, the three-dimensional point coordinates after rotation transformation are translated and corrected to obtain the three-dimensional point coordinates in the visible light module camera coordinate system. The three-dimensional point coordinates in the camera coordinate system of the visible light module are projected onto the pixel coordinate system of the visible light module through the intrinsic parameter matrix of the visible light module to obtain the registered pixel coordinates, thereby generating the registered image.

[0010] In some embodiments, calculating the overlapping region of the registered images of each module, based on the frame of the visible light module, includes: Obtain the set of pixel coordinates of each registered image in the visible light module pixel coordinate system, and extract the minimum and maximum values ​​of the horizontal coordinate and the minimum and maximum values ​​of the vertical coordinate of each module pixel. The maximum value among the minimum x-coordinates of each module is taken as the x-coordinate of the left boundary of the overlapping area; The minimum value among the maximum values ​​of the x-coordinates of each module is taken as the x-coordinate of the right boundary of the overlapping area; The maximum value among the minimum ordinate values ​​of each module is taken as the ordinate of the upper boundary of the overlapping area; The minimum value among the maximum values ​​of the ordinates of each module is taken as the ordinate of the lower boundary of the overlapping area.

[0011] In some embodiments, the boundary completion of the non-overlapping regions using an interpolation method based on distance decay weights includes: For each pixel in the non-overlapping region, calculate the distance from the current pixel coordinates to the boundary of the effective region of each module in the visible light module pixel coordinate system; The completion weights of each module are calculated based on a negative exponential function of the distance. The pixel values ​​of each module at the current pixel coordinates are weighted and summed according to the padding weight to obtain the pixel values ​​after boundary padding.

[0012] In some embodiments, calculating the dynamic fusion weight for each pixel in the image of each module based on pixel gradient entropy and the correlation coefficient between each module and the visible light module includes: For each pixel coordinate in the visible light module pixel coordinate system, a neighborhood window is defined with the current pixel coordinate as the center, and the gradient entropy of each module image within the neighborhood window is calculated. The correlation coefficient between the image of each module and the image of the visible light module within the neighborhood window is obtained by calculating the ratio of the product of the covariance of the pixel values ​​of each module and the visible light module within the neighborhood window to their respective standard deviations. The gradient entropy and the correlation coefficient are weighted and combined according to a preset ratio to obtain the dynamic fusion weight of each module at the current pixel coordinate.

[0013] Secondly, embodiments of this application provide a UAV four-light camera module image fusion system, the system comprising: The preprocessing module is used to calibrate the infrared module, ultraviolet module, visible light module and laser module, obtain the intrinsic and extrinsic parameters of each module, and perform distortion correction and focal length normalization processing on the original images of each module based on the intrinsic parameters to obtain the corrected infrared image, ultraviolet image, visible light image and laser image. The registration module is used to map the corrected infrared image, ultraviolet image and laser image to the visible light module coordinate system, based on the visible light module and combined with the external parameters and the fault point distance information measured in real time by the laser module, to obtain the registered images of each module. The image group acquisition module is used to calculate the overlapping area of ​​the registered images of each module based on the image frame of the visible light module, and to perform boundary filling for the non-overlapping areas using an interpolation method based on distance attenuation weight, so as to obtain the target image group. The fusion module is used to calculate dynamic fusion weights for each pixel in the image of each module based on pixel gradient entropy and the correlation coefficient between each module and the visible light module, and to perform weighted fusion on the target image group according to the dynamic fusion weights to obtain a fused image.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the UAV four-light camera module image fusion method as described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the UAV four-light camera module image fusion method as described in the first aspect above.

[0016] Compared to related technologies, the UAV four-light camera module image fusion method provided in this application first performs intrinsic and extrinsic parameter calibration on the four major modules of infrared, ultraviolet, visible light, and laser. Then, distortion correction and focal length normalization are completed based on the intrinsic parameters. This not only eliminates the influence of lens distortion (such as barrel and pincushion distortion) of each module, but also unifies the focal length specifications of each module, avoiding initial image offset caused by differences in module parameters, and laying a precise foundation for subsequent registration and fusion.

[0017] Using the visible light module, which boasts high image clarity and strong scene recognition, as the registration benchmark, and combining external parameters and real-time fault point distance information measured by the laser module, the images of the other three types of modules are accurately mapped to the visible light coordinate system. The introduction of laser distance information solves the problem of registration offset in long-distance and complex scenes, ensuring that the images of multiple modules are completely aligned in spatial position, thus improving the core pain points of image offset and misalignment in fused images.

[0018] The adaptive adjustment of dynamic weights in the real-time distance feedback of the laser module can cope with complex working conditions in high-altitude and mobile shooting of drones, ensuring stable and accurate fused images. It can be directly applied to practical scenarios such as drone fault detection and inspection, solving the problems of poor adaptability and insufficient practicality of fused images in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a UAV four-light camera module image fusion method according to an embodiment of this application; Figure 2 This is a structural block diagram of a drone four-light camera module image fusion system according to an embodiment of this application; Figure 3 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0021] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0022] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0023] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0024] This embodiment provides a method for image fusion using a four-light camera module for unmanned aerial vehicles (UAVs). Figure 1 This is a flowchart of the UAV four-light camera module image fusion method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: The infrared module, ultraviolet module, visible light module and laser module are calibrated to obtain the intrinsic and extrinsic parameters of each module. Based on the intrinsic parameters, the original images of each module are subjected to distortion correction and focal length normalization processing to obtain the corrected infrared image, ultraviolet image, visible light image and laser image.

[0025] In some embodiments, intrinsic parameters include the effective focal length of each module, the principal point coordinates of the image, the radial distortion coefficient, and the tangential distortion coefficient; extrinsic parameters include the rotation matrix and translation vector of each module relative to the visible light module.

[0026] Optionally, a combination of "Zhang Zhengyou calibration method + laser-assisted hand-eye calibration" can be used to obtain single-module intrinsic parameters and cross-module extrinsic parameters respectively.

[0027] Example of intrinsic parameter calibration: Fix the checkerboard calibration board in a simulated power grid line scenario (5-50m away from the module, covering commonly used inspection distances). Take 20 sets of calibration images from different angles for each module, and calculate the intrinsic parameter matrix K through corner detection. i(i=1,2,3,4 correspond to infrared, ultraviolet, visible light, and laser light, respectively)

[0028] f i The effective focal length (in mm) of the i-th module, such as visible light. f 3=8mm, infrared f 1 = 10 mm; (c x,i c y,i ) represents the coordinates (in pixels) of the principal point of the image of the i-th module, which is the intersection of the lens optical axis and the imaging plane.

[0029] Simultaneously calculate the radial distortion coefficient (k) 1,i k 2,i This refers to correcting pixel stretching at the lens edges, with a value range of []. [0.05, 0.05]; and tangential distortion coefficient (p 1,i p 2,i This refers to correcting distortion caused by lens mounting tilt, with a value range of []. [0.01, 0.01] are used for subsequent distortion correction.

[0030] Example of extrinsic parameter calibration: Using the visible light module as a reference (clear image quality, rich details, suitable as a registration reference), measure the relative positions of the other three modules with the visible light module using the laser module to obtain the extrinsic parameters (rotation matrix Ri, translation vector Ti):

[0031] Where Δx i ,Δy i ,Δz i These represent the X / Y / Z axis distances (in mm) of the i-th module relative to the visible light module, obtained by fitting laser ranging data.

[0032] It should be noted that R i Ti is a 3×3 rotation matrix (describing the attitude angle between modules), and Ti is a 3×1 translation vector (describing the spatial distance between modules).

[0033] The intrinsic parameters (focal length, distortion coefficient) and extrinsic parameters (relative positional relationship) of the four-light module were collected through calibration experiments to provide data support for subsequent calibration.

[0034] In some embodiments, step S101, which involves performing distortion correction and focal length normalization processing on the original images of each module based on intrinsic parameters, includes: Step S1011: Convert the pixel coordinates in the original image pixel coordinate system to normalized coordinates in the normalized image coordinate system.

[0035] Step S1012: Correct the distortion of the normalized coordinates based on the radial distortion coefficient and the tangential distortion coefficient to obtain distortion-free normalized coordinates.

[0036] Step S1013: Backproject the distortion-free normalized coordinates to the pixel coordinate system to obtain the distortion-free pixel coordinates.

[0037] Step S1014: Based on the effective focal length of the visible light module, calculate the ratio of the effective focal length of each module to the effective focal length of the visible light module, and multiply the distortion-free pixel coordinates of each module by the ratio to normalize the focal length of all module images.

[0038] For the original image pixels (u raw v raw Distortion correction is performed to obtain distortion-free pixels (u). undist v undist The distortion correction formula is:

[0039] The focal lengths of all modules are unified and normalized to the focal lengths of the visible light modules. f 3. To avoid image scaling shifts caused by focal length differences, the normalization formula is:

[0040] First, the lens optical distortion (such as edge stretching of infrared lenses) is eliminated by using distortion correction formulas. Then, the scaling ratio of infrared, ultraviolet and laser images is adjusted to be consistent with that of visible light by using focal length normalization, so as to ensure that objects at the same distance are the same size in each image.

[0041] Step S102: Using the visible light module as a reference, and combining the external parameters and the fault point distance information measured in real time by the laser module, the corrected infrared image, ultraviolet image and laser image are mapped to the visible light module coordinate system to obtain the registered images of each module.

[0042] In some embodiments, step S102 specifically includes: Step S1021: The pixel coordinates of the corrected infrared image, ultraviolet image and laser image are transformed into the camera coordinate system of the corresponding module through the inverse matrix of the intrinsic parameter matrix of the corresponding module, so as to obtain the three-dimensional point coordinates in the camera coordinate system of each module.

[0043] Step S1022: Rotate the three-dimensional point coordinates according to the rotation matrix to transform them into an intermediate coordinate system that is aligned with the orientation of the visible light module.

[0044] Step S1023: Based on the translation vector and the fault point distance measured in real time by the laser module, the three-dimensional point coordinates after rotation transformation are translated and corrected to obtain the three-dimensional point coordinates in the visible light module camera coordinate system.

[0045] Step S1024: Project the three-dimensional point coordinates in the camera coordinate system of the visible light module to the pixel coordinate system of the visible light module through the intrinsic parameter matrix of the visible light module to obtain the registered pixel coordinates, thereby generating the registered image.

[0046] Homogeneous coordinate transformation is used to achieve cross-module pixel mapping. Let the pixel coordinates of the visible light module be (u3, v3), and the registered pixels of other modules be (u3, v3). , )satisfy:

[0047] Among them, K i -1 Z is the inverse matrix of the intrinsic parameter matrix of the i-th module, used to convert pixel coordinates into three-dimensional points in the camera coordinate system; Z is the distance to the fault point measured by the laser module (unit: m), which is obtained in real time through laser ranging data and is used to correct the scale error of the translation vector.

[0048] If the registered pixels ( , For pixels that extend beyond the visible light image boundary, bilinear interpolation is used to supplement the pixel values.

[0049] This embodiment introduces a laser distance-based dynamic correction translation vector Z to solve the problem of long-distance offset caused by ignoring distance changes in traditional registration (e.g., at a distance of 50m, the registration error is reduced from 8px to less than 1px). Multi-module image registration is achieved through rigorous 3D coordinate transformation, resulting in high geometric accuracy and fundamentally solving the problems of image offset and misalignment. Real-time laser ranging is introduced for translation correction, overcoming the limitations of traditional planar assumptions and significantly improving the robustness of registration in dynamic and long-distance scenarios. All module images are uniformly mapped to the visible light coordinate system, providing a precise alignment foundation for subsequent fusion and adapting to practical application scenarios such as UAV inspection.

[0050] Step S103: Using the image frame of the visible light module as a reference, calculate the overlapping area of ​​the registered images of each module, and use an interpolation method based on distance attenuation weight to fill in the boundaries of the non-overlapping areas to obtain the target image group.

[0051] In some embodiments, step S103, using the image frame of the visible light module as a reference, calculates the overlapping area of ​​the registered images of each module, including: Step S1031: Obtain the set of pixel coordinates of each registered image in the visible light module pixel coordinate system, and extract the minimum and maximum values ​​of the horizontal coordinate and the vertical coordinate of each module pixel.

[0052] For the differences in image size among the four light modules (e.g., 1920×1080 for ultraviolet image and 2560×1440 for visible light image), the effective overlapping area is calculated and the boundary is adaptively filled to avoid misalignment of the image boundary after fusion.

[0053] Step S1032: Take the maximum value among the minimum values ​​of the abscissas of each module as the abscissa of the left boundary of the overlapping area; take the minimum value among the maximum values ​​of the abscissas of each module as the abscissa of the right boundary of the overlapping area; take the maximum value among the minimum values ​​of the ordinates of each module as the ordinate of the upper boundary of the overlapping area; take the minimum value among the maximum values ​​of the ordinates of each module as the ordinate of the lower boundary of the overlapping area.

[0054] Overlap area calculation, assuming the image area after registration of each module is S i =[W i H i (Width × Height, unit: px), using the visible light frame size S3 = [W3, H3] as a reference, calculate the top left corner (x) of the overlapping area. min y min ) and bottom right corner (x max y max ):

[0055] In some embodiments, step S103, which involves using an interpolation method based on distance attenuation weights to fill in the boundaries of non-overlapping regions, includes: Step S1033: For each pixel in the non-overlapping region, calculate the distance from the current pixel coordinates to the boundary of the effective region of each module in the visible light module pixel coordinate system.

[0056] Step S1034: Calculate the completion weights of each module based on the negative exponential function of distance.

[0057] Step S1035: The pixel values ​​of each module at the current pixel coordinates are weighted and summed according to the padding weight to obtain the pixel values ​​after boundary padding.

[0058] For non-overlapping regions, modal feature interpolation is used to fill in the gaps, as shown in the following formula:

[0059] Among them, supplementing the weight d j Let I be the distance (in pixels) from pixel (x, y) to the effective area of ​​the j-th module, where α = 0.1 is the attenuation coefficient.j (x,y) represents the pixel value of the j-th module.

[0060] By calculating the common field of view of the four-light images through overlapping regions, and fusing the pixel features of each module based on distance attenuation weights in non-overlapping regions, the boundary transition is ensured to be natural and without obvious misalignment (boundary alignment error ≤ 1px).

[0061] Based on the visible light frame, the overlapping area of ​​each module after registration is accurately calculated. At the same time, distance attenuation weighted interpolation is used to fill in the boundaries of non-overlapping areas. This avoids black edges and breaks in non-overlapping areas, while ensuring natural boundary transitions. The merged image fully covers the shooting range of each module without losing any scene details (such as distant fault points and infrared / ultraviolet features in edge areas).

[0062] Step S104: Based on the pixel gradient entropy and the correlation coefficient between each module and the visible light module, calculate the dynamic fusion weight for each pixel in the image of each module, and perform weighted fusion on the target image group according to the dynamic fusion weight to obtain the fused image.

[0063] By combining the characteristics of power grid fault detection (such as infrared-outer heavy thermal defects and ultraviolet-outer heavy corona), an adaptive weight fusion algorithm is adopted to achieve complementary optimization of multimodal information while maintaining consistency between content and boundaries. Dynamic weight fusion takes into account the modal specificity of power grid faults (such as increasing the ultraviolet weight in the corona discharge region and increasing the infrared weight in the thermal defect region), solving the feature loss problem caused by traditional fixed weight fusion.

[0064] In some embodiments, step S104, which calculates the dynamic fusion weight for each pixel in the image of each module based on the pixel gradient entropy and the correlation coefficient between each module and the visible light module, includes: Step S1041: For each pixel coordinate in the visible light module pixel coordinate system, a neighborhood window is defined with the current pixel coordinate as the center, and the gradient entropy of each module image within the neighborhood window is calculated. Step S1042: By calculating the ratio of the product of the covariance of the pixel values ​​of each module and the visible light module within the neighborhood window to their respective standard deviations, the correlation coefficient between the images of each module and the visible light module within the neighborhood window is obtained. Step S1043: The gradient entropy and correlation coefficient are weighted and combined according to a preset proportional coefficient to obtain the dynamic fusion weight of each module at the current pixel coordinate.

[0065] Dynamic weights w are calculated based on pixel gradient entropy and modal correlation. i (x,y):

[0066] in, p (g ) represents the gradient histogram probability of the (x,y) neighborhood (3×3) of pixel; Corr(I i ,I3) is the correlation coefficient between the i-th module and the visible light module; optionally, β=0.6 is the weighting coefficient (balancing detailed features and correlation).

[0067] The fusion formula is:

[0068] in, Let I be the pixel value of the i-th module after registration and boundary alignment. fusion (x,y) represents the dynamic fusion weights.

[0069] Based on pixel gradient entropy (reflecting pixel detail richness) and the correlation coefficient between each module and the visible light module, a dynamic fusion weight is calculated for each pixel—pixels with rich detail and high correlation are assigned higher weights, while those with less detail and lower correlation are assigned lower weights. Compared to fixed-weight fusion, this method can adapt to different scenes (such as strong light, weak light, and fault areas), allowing the fused image to retain both the clear scene outline of the visible light and highlight the abnormal features of infrared and ultraviolet light, as well as the distance information of lasers, thus balancing clarity and feature recognition.

[0070] Through the above steps, the intrinsic and extrinsic parameters of the four major modules—infrared, ultraviolet, visible light, and laser—are first calibrated. Then, distortion correction and focal length normalization are completed based on the intrinsic parameters. This not only eliminates the influence of lens distortion (such as barrel and pincushion distortion) of each module, but also unifies the focal length specifications of each module, avoiding initial image offset caused by differences in module parameters, and laying a precise foundation for subsequent registration and fusion.

[0071] Using the visible light module, which boasts high image clarity and strong scene recognition, as the registration benchmark, and combining external parameters and real-time fault point distance information measured by the laser module, the images of the other three types of modules are accurately mapped to the visible light coordinate system. The introduction of laser distance information solves the problem of registration offset in long-distance and complex scenes, ensuring that the images of multiple modules are completely aligned in spatial position, thus improving the core pain points of image offset and misalignment in fused images.

[0072] The adaptive adjustment of dynamic weights in the real-time distance feedback of the laser module can cope with complex working conditions in high-altitude and mobile shooting of drones, ensuring stable and accurate fused images. It can be directly applied to practical scenarios such as drone fault detection and inspection, solving the problems of poor adaptability and insufficient practicality of fused images in related technologies.

[0073] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0074] This embodiment also provides a UAV four-light camera module image fusion system, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0075] Figure 2 This is a structural block diagram of a drone four-light camera module image fusion system according to an embodiment of this application, such as... Figure 2 As shown, the system includes: The preprocessing module 21 is used to calibrate the infrared module, ultraviolet module, visible light module and laser module, obtain the intrinsic and extrinsic parameters of each module, and perform distortion correction and focal length normalization processing on the original images of each module based on the intrinsic parameters to obtain the corrected infrared image, ultraviolet image, visible light image and laser image.

[0076] The registration module 22 is used to map the corrected infrared image, ultraviolet image and laser image to the visible light module coordinate system, based on the visible light module and combined with the external parameters and the fault point distance information measured in real time by the laser module, so as to obtain the registered images of each module.

[0077] The image group acquisition module 23 is used to calculate the overlapping area of ​​the registered images of each module based on the frame of the visible light module, and to perform boundary filling for the non-overlapping areas using an interpolation method based on distance attenuation weight, so as to obtain the target image group.

[0078] The fusion module 24 is used to calculate the dynamic fusion weight for each pixel in the image of each module based on the pixel gradient entropy and the correlation coefficient between each module and the visible light module, and to perform weighted fusion on the target image group according to the dynamic fusion weight to obtain the fused image.

[0079] In some embodiments, intrinsic parameters include the effective focal length of each module, the principal point coordinates of the image, the radial distortion coefficient, and the tangential distortion coefficient; extrinsic parameters include the rotation matrix and translation vector of each module relative to the visible light module.

[0080] In some embodiments, the preprocessing module 21 includes: The coordinate normalization module is used to convert the pixel coordinates in the original image pixel coordinate system into normalized coordinates in the normalized image coordinate system.

[0081] The correction module is used to correct the distortion of the normalized coordinates based on the radial distortion coefficient and the tangential distortion coefficient, so as to obtain distortion-free normalized coordinates.

[0082] The back projection module is used to backproject distortion-free normalized coordinates to the pixel coordinate system to obtain distortion-free pixel coordinates.

[0083] The focal length normalization module is used to calculate the ratio of the effective focal length of each module to the effective focal length of the visible light module, based on the effective focal length of the visible light module, and multiply the distortion-free pixel coordinates of each module by the ratio to achieve focal length normalization of all module images.

[0084] In some embodiments, the registration module 22 includes: The coordinate system transformation module is used to transform the pixel coordinates of the calibrated infrared, ultraviolet and laser images to the camera coordinate system of the corresponding module through the inverse matrix of the intrinsic parameter matrix of the module, so as to obtain the three-dimensional point coordinates in the camera coordinate system of each module.

[0085] The attitude alignment module is used to rotate and transform the coordinates of three-dimensional points according to the rotation matrix, so that they are transformed into an intermediate coordinate system that is aligned with the attitude of the visible light module.

[0086] The translation correction module is used to perform translation correction on the three-dimensional point coordinates after rotation transformation based on the translation vector and the fault point distance measured in real time by the laser module, so as to obtain the three-dimensional point coordinates in the visible light module camera coordinate system.

[0087] The pixel coordinate registration module is used to project the coordinates of three-dimensional points in the camera coordinate system of the visible light module to the pixel coordinate system of the visible light module through the intrinsic parameter matrix of the visible light module, so as to obtain the registered pixel coordinates and generate the registered image.

[0088] In some embodiments, the image group acquisition module 23 includes: The maximum / minimum value acquisition module is used to obtain the set of pixel coordinates of each module's registered image in the visible light module pixel coordinate system, and extract the minimum and maximum values ​​of the horizontal coordinates and the vertical coordinates of each module's pixels.

[0089] The region determination module is used to take the maximum value among the minimum x-coordinates of each module as the x-coordinate of the left boundary of the overlapping region; take the minimum value among the maximum x-coordinates of each module as the x-coordinate of the right boundary of the overlapping region; take the maximum value among the minimum y-coordinates of each module as the y-coordinate of the upper boundary of the overlapping region; and take the minimum value among the maximum y-coordinates of each module as the y-coordinate of the lower boundary of the overlapping region.

[0090] In some embodiments, the image group acquisition module 23 includes: The distance calculation module is used to calculate the distance from the current pixel coordinates to the boundary of the effective area of ​​each module for each pixel in the visible light module pixel coordinate system, for each pixel in the non-overlapping area.

[0091] The completion weight determination module is used to calculate the completion weight of each module based on a negative exponential function of distance.

[0092] The boundary completion module is used to sum the pixel values ​​of each module at the current pixel coordinates according to the completion weight, so as to obtain the pixel value after boundary completion.

[0093] In some embodiments, the fusion module 24 includes: The gradient entropy calculation module is used to calculate the gradient entropy of each module image within the neighborhood window centered on the current pixel coordinates for each pixel coordinate in the visible light module pixel coordinate system.

[0094] The correlation coefficient calculation module is used to obtain the correlation coefficient between the image of each module and the image of the visible light module within the neighborhood window by calculating the ratio of the product of the covariance of the pixel values ​​of each module and the visible light module within the neighborhood window to their respective standard deviations.

[0095] The fusion weight determination module is used to weight and combine the gradient entropy and correlation coefficient according to a preset ratio to obtain the dynamic fusion weight of each module at the current pixel coordinate.

[0096] The above system first calibrates the intrinsic and extrinsic parameters of the four modules: infrared, ultraviolet, visible light, and laser. Then, based on the intrinsic parameters, distortion correction and focal length normalization are completed. This not only eliminates the influence of lens distortion (such as barrel and pincushion distortion) of each module, but also unifies the focal length specifications of each module, avoiding initial image offset caused by differences in module parameters, and laying a precise foundation for subsequent registration and fusion.

[0097] Using the visible light module, which boasts high image clarity and strong scene recognition, as the registration benchmark, and combining external parameters and real-time fault point distance information measured by the laser module, the images of the other three types of modules are accurately mapped to the visible light coordinate system. The introduction of laser distance information solves the problem of registration offset in long-distance and complex scenes, ensuring that the images of multiple modules are completely aligned in spatial position, thus improving the core pain points of image offset and misalignment in fused images.

[0098] The adaptive adjustment of dynamic weights in the real-time distance feedback of the laser module can cope with complex working conditions in high-altitude and mobile shooting of drones, ensuring stable and accurate fused images. It can be directly applied to practical scenarios such as drone fault detection and inspection, solving the problems of poor adaptability and insufficient practicality of fused images in related technologies.

[0099] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0100] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0101] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0102] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1 calibrates the infrared module, ultraviolet module, visible light module and laser module, obtains the intrinsic and extrinsic parameters of each module, and performs distortion correction and focal length normalization processing on the original images of each module based on the intrinsic parameters to obtain the corrected infrared image, ultraviolet image, visible light image and laser image.

[0103] S2, using the visible light module as a reference, combines external parameters and the fault point distance information measured in real time by the laser module to map the corrected infrared image, ultraviolet image and laser image onto the visible light module coordinate system, thus obtaining the registered images of each module.

[0104] S3, using the image frame of the visible light module as a reference, calculates the overlapping area of ​​the registered images of each module, and uses an interpolation method based on distance attenuation weight to fill in the boundaries of the non-overlapping areas, thus obtaining the target image group.

[0105] S4. Based on the pixel gradient entropy and the correlation coefficient between each module and the visible light module, calculate the dynamic fusion weight for each pixel in the image of each module, and perform weighted fusion on the target image group according to the dynamic fusion weight to obtain the fused image.

[0106] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0107] In one embodiment, Figure 3 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 3 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 3As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for image fusion using a four-light camera module for unmanned aerial vehicles (UAVs).

[0108] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0110] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for image fusion using a four-light camera module of a drone, characterized in that, The method includes: The infrared module, ultraviolet module, visible light module and laser module are calibrated to obtain the intrinsic and extrinsic parameters of each module. Based on the intrinsic parameters, the original images of each module are subjected to distortion correction and focal length normalization to obtain the corrected infrared image, ultraviolet image, visible light image and laser image. Using the visible light module as a reference, and combining the external parameters and the fault point distance information measured in real time by the laser module, the corrected infrared image, ultraviolet image and laser image are mapped to the visible light module coordinate system to obtain the registered images of each module; Using the image frame of the visible light module as a reference, the overlapping area of ​​the registered images of each module is calculated, and the non-overlapping area is filled in by an interpolation method based on distance attenuation weight to obtain the target image group. Based on pixel gradient entropy and the correlation coefficient between each module and the visible light module, a dynamic fusion weight is calculated for each pixel in the image of each module, and the target image group is weighted and fused according to the dynamic fusion weight to obtain a fused image.

2. The method according to claim 1, characterized in that, The intrinsic parameters include the effective focal length of each module, the coordinates of the principal point of the image, the radial distortion coefficient, and the tangential distortion coefficient; the extrinsic parameters include the rotation matrix and translation vector of each module relative to the visible light module.

3. The method according to claim 2, characterized in that, The process of performing distortion correction and focal length normalization on the original images of each module based on the intrinsic parameters includes: Convert the pixel coordinates in the original image pixel coordinate system to normalized coordinates in the normalized image coordinate system; The normalized coordinates are corrected for distortion based on the radial distortion coefficient and the tangential distortion coefficient to obtain distortion-free normalized coordinates; The distortion-free normalized coordinates are back-projected onto the pixel coordinate system to obtain the distortion-free pixel coordinates; Based on the effective focal length of the visible light module, the ratio of the effective focal length of each module to the effective focal length of the visible light module is calculated, and the distortion-free pixel coordinates of each module are multiplied by this ratio to normalize the focal length of all module images.

4. The method according to claim 3, characterized in that, The process of mapping the corrected infrared, ultraviolet, and laser images onto the visible light module coordinate system to obtain the registered images of each module includes: The pixel coordinates of the corrected infrared, ultraviolet and laser images are transformed into the camera coordinate system of the corresponding module through the inverse matrix of the intrinsic parameter matrix of the module, so as to obtain the three-dimensional point coordinates in the camera coordinate system of each module. The three-dimensional point coordinates are rotated and transformed according to the rotation matrix to an intermediate coordinate system aligned with the orientation of the visible light module; Based on the translation vector and the fault point distance measured in real time by the laser module, the three-dimensional point coordinates after rotation transformation are translated and corrected to obtain the three-dimensional point coordinates in the visible light module camera coordinate system. The three-dimensional point coordinates in the camera coordinate system of the visible light module are projected onto the pixel coordinate system of the visible light module through the intrinsic parameter matrix of the visible light module to obtain the registered pixel coordinates, thereby generating the registered image.

5. The method according to claim 1, characterized in that, The calculation of the overlapping area of ​​the registered images of each module, based on the image frame of the visible light module, includes: Obtain the set of pixel coordinates of each registered image in the visible light module pixel coordinate system, and extract the minimum and maximum values ​​of the horizontal coordinate and the minimum and maximum values ​​of the vertical coordinate of each module pixel. The maximum value among the minimum x-coordinates of each module is taken as the x-coordinate of the left boundary of the overlapping area; The minimum value among the maximum values ​​of the x-coordinates of each module is taken as the x-coordinate of the right boundary of the overlapping area; The maximum value among the minimum ordinate values ​​of each module is taken as the ordinate of the upper boundary of the overlapping area; The minimum value among the maximum values ​​of the ordinates of each module is taken as the ordinate of the lower boundary of the overlapping area.

6. The method according to claim 1, characterized in that, The method of using an interpolation method based on distance decay weights to fill in the boundaries of non-overlapping regions includes: For each pixel in the non-overlapping region, calculate the distance from the current pixel coordinates to the boundary of the effective region of each module in the visible light module pixel coordinate system; The completion weights of each module are calculated based on a negative exponential function of the distance. The pixel values ​​of each module at the current pixel coordinates are weighted and summed according to the padding weight to obtain the pixel values ​​after boundary padding.

7. The method according to claim 1, characterized in that, The calculation of dynamic fusion weights for each pixel in the image of each module, based on pixel gradient entropy and the correlation coefficient between each module and the visible light module, includes: For each pixel coordinate in the visible light module pixel coordinate system, a neighborhood window is defined with the current pixel coordinate as the center, and the gradient entropy of each module image within the neighborhood window is calculated. The correlation coefficient between the image of each module and the image of the visible light module within the neighborhood window is obtained by calculating the ratio of the product of the covariance of the pixel values ​​of each module and the visible light module within the neighborhood window to their respective standard deviations. The gradient entropy and the correlation coefficient are weighted and combined according to a preset ratio to obtain the dynamic fusion weight of each module at the current pixel coordinate.

8. A UAV four-light camera module image fusion system, characterized in that, The system includes: The preprocessing module is used to calibrate the infrared module, ultraviolet module, visible light module and laser module, obtain the intrinsic and extrinsic parameters of each module, and perform distortion correction and focal length normalization processing on the original images of each module based on the intrinsic parameters to obtain the corrected infrared image, ultraviolet image, visible light image and laser image. The registration module is used to map the corrected infrared image, ultraviolet image and laser image to the visible light module coordinate system, based on the visible light module and combined with the external parameters and the fault point distance information measured in real time by the laser module, to obtain the registered images of each module. The image group acquisition module is used to calculate the overlapping area of ​​the registered images of each module based on the image frame of the visible light module, and to perform boundary filling for the non-overlapping areas using an interpolation method based on distance attenuation weight, so as to obtain the target image group. The fusion module is used to calculate dynamic fusion weights for each pixel in the image of each module based on pixel gradient entropy and the correlation coefficient between each module and the visible light module, and to perform weighted fusion on the target image group according to the dynamic fusion weights to obtain a fused image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the UAV four-light camera module image fusion method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the UAV four-light camera module image fusion method as described in any one of claims 1 to 7.

Images