A multispectral fusion imaging camera module and image processing method

Abstract

The application relates to the technical field of image processing, and discloses a multispectral fusion imaging camera module and an image processing method, which comprises the following steps: performing spatial mapping on infrared imaging images and near-infrared imaging images based on the spatial arrangement order of corner points and the direction of continuous edge broken lines; determining a spectral corresponding structure according to the light and dark trend changes of the same scene in three imaging channels; performing regional division on a geometric alignment result based on the spectral corresponding structure; performing direction consistency correction on geometric edges in a visible light imaging image based on a dominant contour line; performing extensibility superposition on texture lines in the spatially mapped near-infrared imaging image based on the texture direction of a texture detail area; performing superposition on the brightness levels of the visible light imaging image based on the brightness extension direction of a background smooth area; and performing connectivity checking on the boundary direction, the texture direction and the brightness extension direction. The application ensures the reliability of multispectral image fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multispectral fusion imaging camera module and an image processing method. BACKGROUND

[0002] Multispectral fusion imaging is a technology that can capture the reflection or radiation information of an object in multiple different wavelength spectral bands, which is widely used in many fields such as security monitoring, agricultural detection, medical diagnosis, and environmental monitoring.

[0003] Chinese Patent Publication No. CN120164068A discloses an infrared and visible light image fusion method based on deep learning, including the following steps: S1. Image preprocessing: preprocessing the input infrared and visible light images, including noise removal and contrast enhancement; S2. Feature extraction: extracting the features of the infrared and visible light images using CNN; S3. Feature fusion: fusing the obtained features respectively; S4. Image reconstruction: reconstructing the fused feature images into a fusion image; S5. Target detection and evaluation: detecting the target in the fused image and evaluating the accuracy. As can be seen, it does not perform differential fusion according to the characteristics of different regions of the image, and only pre-processes the image without establishing geometric constraints based on corner points and edge fold lines, which is prone to local image distortion, such as edge misalignment between infrared and visible light images, thereby affecting the fusion result.

[0004] Therefore, it is necessary to design a multispectral fusion imaging camera module and an image processing method to solve the problems existing in the current technology. SUMMARY

[0005] In view of this, the present application proposes a multispectral fusion imaging camera module and an image processing method, aiming to solve the problem that differential fusion cannot be performed according to the characteristics of different regions of the image, and geometric constraints based on corner points and edge fold lines are not established, which is prone to local image distortion, thereby affecting the fusion result.

[0006] In one aspect, the present application proposes an image processing method for a multispectral fusion imaging camera module, comprising:

[0007] Collecting visible light imaging images, infrared imaging images, and near-infrared imaging images of three imaging channels, taking the visible light imaging images as a geometric reference, spatially mapping the infrared imaging images and near-infrared imaging images based on the spatial arrangement order of the corner points and the direction of the continuous edge fold lines, and determining a geometric alignment result;

[0008] According to the light and dark trend changes of the same scene in the three imaging channels, the light and dark extension directions of the high reflection part, the low reflection part and the translucent tissue region are characterized, the spectral corresponding structure is determined, the geometric alignment result is divided into regions based on the spectral corresponding structure, the contour enhancement region, the texture detail region and the background smoothing region are determined;

[0009] The dominant contour line in the spatial mapped infrared imaging image is determined based on the boundary direction of the contour enhancement region, the geometric edge in the visible light imaging image is corrected for direction consistency based on the dominant contour line, the contour fusion result is determined, the texture line in the spatial mapped near-infrared imaging image is extended and superimposed based on the texture direction of the texture detail region, the texture fusion result is determined, and the brightness level of the visible light imaging image is superimposed based on the brightness extension direction of the background smoothing region, and the background fusion result is determined.

[0010] The boundary direction, the texture direction and the brightness extension direction are connected and checked, when any region has a fracture, the boundary direction, the texture direction or the brightness extension direction is re-determined according to the fracture position and backtracking fusion is performed, and when the connection check is passed, the fusion image is output.

[0011] Further, when the visible light imaging image is taken as a geometric reference, the infrared imaging image and the near-infrared imaging image are spatially mapped based on the corner point spatial arrangement order and the continuous edge fold line direction to determine the geometric alignment result, including:

[0012] The corner point spatial arrangement order is determined based on a marker shape constraint framework in the three imaging channels, the geometric reference of the visible light imaging image is determined based on the corner point spatial arrangement order, and the corners of the infrared imaging image and the near-infrared imaging image are sequentially mapped to the geometric reference, and the turning direction of the continuous edge fold line is constrained to determine the geometric alignment result.

[0013] Further, when the light and dark extension directions of the high reflection part, the low reflection part and the translucent tissue region are characterized according to the light and dark trend changes of the same scene in the three imaging channels, the spectral corresponding structure is determined, including:

[0014] The light and dark trend of the same field of view region of the three imaging channels is analyzed, the slow change section, the sudden change section and the continuous growth section of the light and dark trend are identified, and the light and dark change structure is determined.

[0015] The boundary fold line position and the light and dark change structure are corresponded to determine the light and dark transition of different spectral channels at the object boundary, and the light and dark extension directions of each spectral channel at the high reflection part, the low reflection part and the translucent tissue region are recorded, and the spectral corresponding structure is determined based on the light and dark extension directions.

[0016] Further, in the region division of the geometric alignment result based on the spectral corresponding structure, when determining the contour enhancement region, the texture detail region and the background smooth region, comprising:

[0017] Pixel neighborhood traversal is performed on the geometric alignment result.

[0018] The common boundary line segment of the continuous edge broken line is extracted, the pixel number ratio of the common boundary line segment in the pixel neighborhood is determined, and the turning direction deviation of the common boundary line segment in the pixel neighborhood is determined. When the pixel number ratio is greater than or equal to the pixel number ratio threshold value, and the turning direction deviation is less than or equal to the turning direction deviation threshold value, the region where the pixel neighborhood is located is determined as the contour enhancement region.

[0019] The junction line of the slowly varying segment and the continuously increasing segment from the spectral corresponding structure is extracted as the texture junction line, the texture density of the texture junction line in the pixel neighborhood is determined, and the included angle of adjacent texture junction lines is taken as the junction trend deviation. When the texture density is greater than or equal to the texture density threshold value, and the junction trend deviation is less than or equal to the junction trend deviation threshold value, the region where the pixel neighborhood is located is determined as the texture detail region.

[0020] The luminance gradient of the geometric alignment result in the pixel neighborhood is determined, the maximum luminance gradient is determined as the background gradient, the luminance standard deviation of the geometric alignment result in the pixel neighborhood is determined based on the spectral corresponding structure, and when the background gradient is less than or equal to the background gradient threshold value, and the luminance standard deviation is less than or equal to the luminance standard deviation threshold value, the region where the pixel neighborhood is located is determined as the background smooth region.

[0021] Further, in the region division of the geometric alignment result based on the spectral corresponding structure, when determining the contour enhancement region, the texture detail region and the background smooth region, further comprising:

[0022] When the pixel neighborhood meets multiple region conditions at the same time, the region attribution is determined according to the priority of the contour enhancement region, the texture detail region and the background smooth region.

[0023] When the pixel neighborhood does not meet all the region conditions, the adjacent pixel neighborhood which has been determined as the contour enhancement region, the texture detail region or the background smooth region is taken as the adjacent pixel neighborhood, the number of the contour enhancement region, the texture detail region and the background smooth region in the adjacent pixel neighborhood is counted respectively, and the region to which the maximum number belongs is determined as the region attribution of the pixel neighborhood. When the maximum number exists at the same time, the region attribution is determined according to the priority of the contour enhancement region, the texture detail region and the background smooth region.

[0024] Further, in determining the dominant contour line in the spatially mapped infrared imaging image based on the boundary direction of the contour-enhanced region, when performing direction consistency correction on the geometric edge in the visible light imaging image based on the dominant contour line, comprising:

[0025] Taking the high-contrast boundary in the spatially mapped infrared imaging image as the dominant contour line based on the boundary direction of the contour-enhanced region, determining the geometric edge contour line in the visible light imaging image;

[0026] When there is a deviation in the direction of the dominant contour line and the geometric edge contour line, then adjusting the turning direction of the geometric edge contour line, and superimposing the dominant contour line and the adjusted geometric edge contour line in the same direction.

[0027] Further, in extending the texture line in the spatially mapped near-infrared imaging image based on the texture direction of the texture detail region, comprising:

[0028] Determining the texture framework based on the texture direction of all texture detail regions in the visible light imaging image, and taking the texture framework as the reference texture direction;

[0029] Along the reference texture direction, superimposing the texture line in the spatially mapped near-infrared imaging image into the texture framework;

[0030] When the extension direction of the texture line conflicts with the texture framework, then performing direction conversion on the texture line, and superimposing the result of the direction conversion into the texture framework;

[0031] When the extension direction of the texture line does not conflict with the texture framework, then keeping the extension direction of the texture line.

[0032] Further, in superimposing the brightness level of the visible light imaging image based on the brightness extension direction of the background smooth region, comprising:

[0033] Determining the brightness extension direction of the background smooth region, and taking the brightness of the spatially mapped infrared imaging image as the dominant channel, taking the brightness extension direction as the brightness reference of the background smooth region, and performing piecewise superposition on the brightness reference based on the visible light imaging image.

[0034] Further, in checking the connectivity of the boundary direction, the texture direction, and the brightness extension direction, when any region has a break, redetermining the boundary direction, the texture direction, or the brightness extension direction according to the break position and performing backtracking fusion, comprising:

[0035] The boundary broken line of the contour enhancement area is connected and detected, if the boundary broken line is broken, the turning direction of the boundary broken line at the broken point is re-determined, and the direction consistency correction is re-performed;

[0036] The texture direction of the texture detail area is continuously detected, if the texture frame has a direction mutation, the texture direction of the texture frame with the direction mutation is converted, and the extension superposition is re-performed.

[0037] The brightness extension direction of the background smooth area is smoothly detected, if the brightness extension direction is broken, the brightness extension direction is re-determined according to the position of the broken point, and the superposition is re-performed.

[0038] Compared with the prior art, the beneficial effects of the present application are that: by collecting visible light imaging images, infrared imaging images and near-infrared imaging images of three imaging channels, taking the visible light imaging image as a geometric reference, combining the spatial arrangement order of the corner points and the direction of the continuous edge broken line for spatial mapping, the position and edge direction of the infrared imaging image and the near-infrared imaging image are simultaneously constrained from the global and local levels, the risk of local image distortion and edge misplacement caused by simple registration is avoided, the light and dark extension directions are represented based on the light and dark trend changes of the same scene, the spectral corresponding structure is determined, and the contour enhancement area, the texture detail area and the background smooth area are divided, so that the fusion strategy of each area can adapt to its own characteristic attributes. The boundary direction of the contour enhancement area is used to determine the dominant contour line of the infrared imaging image, the geometric edge of the visible light imaging image is corrected, and the contour clarity of the fusion process is ensured, the texture direction of the texture detail area is used to superimpose the texture lines of the near-infrared imaging image, so as to enrich the details of the texture level, the brightness extension direction of the background smooth area is used, and the smoothness of the background and the hierarchy during fusion are considered, so as to ensure the continuity of the fused image: through the connectivity check of the boundary direction, the texture direction and the brightness extension direction and the backtracking fusion, the risk of artifacts and discontinuity in the fused image can be avoided, and the reliability of the fusion result is improved.

[0039] On the other hand, the present application also provides a multi-spectral fusion imaging camera module for applying the image processing method of the multi-spectral fusion imaging camera module, comprising:

[0040] A camera lens, a camera block and a camera body;

[0041] The camera lens is used for shooting visible light imaging images, infrared imaging images and near-infrared imaging images;

[0042] One end of the camera block is connected to the camera lens, and the other end of the camera block is connected to the camera body.

[0043] It can be understood that the above-mentioned multispectral fusion imaging camera module and image processing method have the same beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0045] Figure 1 A flow chart of an image processing method of a multispectral fusion imaging camera module provided by an embodiment of the present application;

[0046] Figure 2 A structural schematic diagram of a multispectral fusion imaging camera module provided by an embodiment of the present application.

[0047] In the figure: 1, camera lens; 2, camera block; 3, camera body. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0050] Reference Figure 1 As shown in the figure, in some embodiments of the present application, an image processing method of a multispectral fusion imaging camera module comprises:

[0051] S100: Collecting visible light imaging images, infrared imaging images and near-infrared imaging images of three imaging channels, taking the visible light imaging images as a geometric reference, performing spatial mapping on the infrared imaging images and the near-infrared imaging images based on the spatial arrangement order of the corner points and the direction of the continuous edge fold line, and determining a geometric alignment result.

[0052] S200: According to the light and dark trend changes in the three imaging channels of the same scene, the light and dark extension direction of the high reflection part, the low reflection part and the translucent tissue region is characterized, the spectral corresponding structure is determined, the geometric alignment result is regionally divided based on the spectral corresponding structure, the contour enhancement area, the texture detail area and the background smoothing area are determined.

[0053] S300: Based on the boundary direction of the contour enhancement area, the dominant contour line in the spatial mapped infrared imaging image is determined, the geometric edge in the visible light imaging image is direction consistency corrected based on the dominant contour line, the contour fusion result is determined, the texture line in the spatial mapped near-infrared imaging image is extended and superimposed based on the texture direction of the texture detail area, the texture fusion result is determined, the brightness level of the visible light imaging image is superimposed based on the brightness extension direction of the background smoothing area, and the background fusion result is determined.

[0054] S400: The connectivity of the boundary direction, the texture direction and the brightness extension direction is checked, when there is a break in any region, the boundary direction, the texture direction or the brightness extension direction is re-determined according to the break position and backtracking fusion is performed, and when the connectivity check is passed, the fusion image is output.

[0055] Specifically, S100 step firstly collects three imaging channels of visible light imaging image, infrared imaging image and near-infrared imaging image, wherein, the visible light imaging image is designated as a geometric reference, the wavelength range is generally 400-700 nanometers, which can provide high spatial resolution and clear geometric structure closest to human eye vision, so as to be used as a geometric reference, which ensures the reliability of image fusion. The spatial arrangement order of the corner points represents the key corner points in the image, that is, the intersection points of the object contour or the feature points where the curvature changes obviously, such as the corners of buildings or the sharp corners of mechanical parts, the two-dimensional coordinate sequence of which can be extracted by Harris corner point detection algorithm, so as to determine the relative position and topological order in space. The direction of the continuous edge polyline is a continuous boundary path formed by connecting the heads and tails of multiple line segments, and the direction can be determined according to the slope vector of each line segment and the image histogram statistics. The direction of the continuous edge polyline accurately describes the continuity of the object boundary. The spatial mapping transformation is performed by matching the spatial arrangement order of the corner points and the direction of the continuous edge polyline in the visible light imaging image, the infrared imaging image and the near-infrared imaging image, so as to map the pixel coordinates of the infrared imaging image and the near-infrared imaging image to the coordinate system of the visible light imaging image, thereby determining the geometric alignment result, that is, the spatial positions of the visible light imaging image, the infrared imaging image and the near-infrared imaging image are overlapped at the pixel level, which avoids the risk of local image distortion caused by the lack of geometric constraints, for example, the edge misalignment of the infrared imaging image and the visible light imaging image, such as the breakage of the human body contour in the security monitoring, etc. Through the topological preservation of the spatial arrangement order of the corner points and the path continuity constraint of the direction of the continuous edge polyline, the consistency of the multispectral image in the spatial dimension is ensured, and the geometric distortion in the fusion process is avoided. S200 step deeply analyzes the light and dark trend changes of the same scene in three imaging channels, that is, for each pixel position, the trend of the brightness value gradient change in the visible light, infrared and near-infrared wave bands is determined, and the light and dark extension direction of the high reflection part (the object surface reflectivity is relatively high, such as the metal surface or the water surface, which presents high brightness value in the visible light imaging image, but the brightness may be reduced in the infrared imaging image due to the heat radiation characteristics), the low reflection part (the object surface reflectivity is relatively low, such as dark fabric or vegetation shadow, which has low brightness value in the visible light imaging image, but the brightness may be increased in the near-infrared imaging image due to the fluorescence effect of the vegetation), and the semi-transparent tissue region (the region where the light cannot completely penetrate and is not completely blocked due to the composition unevenness or microtopography difference, which presents partial light transmission, partial scattering / refraction optical characteristics, such as semi-transparent ceramics, interlayer gap in 3D printed parts, etc.), that is, the extension trend of the brightness change along a certain spatial direction, so as to determine the spectral corresponding structure. The spectral corresponding structure converts the response difference of different spectral channels to the same object / scene into quantitative and related structure information, which provides a basis for subsequent regional fusion.The geometric alignment result is regionally divided based on the spectral corresponding structure to determine a contour enhancement region (a region where the object boundary is clear and the contour needs to be enhanced), a texture detail region (a region where the surface texture is rich and the details need to be preserved), and a background smoothing region (a region where the background needs to be smoothed). Thus, the different regions of the image are processed differently to ensure that the high reflection, low reflection and semi-transparent regions are optimized in the fusion, and the accuracy and integrity of the fused image are improved.

[0056] It can be understood that, in the S300 step, based on the boundary direction of the contour enhancement region, that is, the trend of the object boundary in the contour enhancement region, the dominant contour line in the spatially mapped infrared imaging image is determined, that is, the contour feature line with high continuity and obvious edge features in the infrared imaging image, the geometric edges in the visible light imaging image are corrected for direction consistency based on the dominant contour line, and thus the contour fusion result, that is, the fusion image of the contour enhancement region, is determined. For the texture detail region, based on the texture direction, that is, the extension direction of the surface texture in the texture detail region, the texture lines in the spatially mapped near-infrared imaging image are extended and superimposed, and the details of the surface texture are expanded, and thus the texture fusion result is determined. For the background smoothing region, based on the brightness extension direction, that is, the dominant direction of the brightness gradient in the background smoothing region, the brightness levels (gradient distribution sequence of the brightness value, such as a smooth transition segment from dark to light) of the visible light imaging image are superimposed to determine the background fusion result. The contour enhancement region eliminates the risk of edge blur through direction consistency correction, the texture detail region retains the details of the texture, such as the texture features of crop lesions in agricultural detection, through extension superposition, and the background smoothing region improves the smoothing effect of the background through brightness level superposition, thereby achieving precise optimization of the contour enhancement region, the texture detail region and the background smoothing region, and improving the visual quality of the fused image. The S400 step checks the connectivity of the boundary direction, the texture direction and the brightness extension direction. When any region is broken (contour enhancement region / texture detail region / background smoothing region), backtracking fusion is performed, that is, only the broken region is re-executed for fusion to avoid the risk of image distortion caused by global recalculation. When the connectivity check passes, it indicates that the fusion process has not been broken, and the fused image is output. The dynamic backtracking mechanism prevents local artifacts caused by breaks, such as jagged distortion of tissue boundaries in medical diagnosis, thereby ensuring the reliability of the fusion result.

[0057] In some embodiments of the present application, when the visible light imaging image is taken as the geometric reference, the infrared imaging image and the near-infrared imaging image are spatially mapped based on the spatial arrangement order of the corner points and the direction of the continuous edge fold line to determine the geometric alignment result, comprising: determining the spatial arrangement order of the corner points based on the marker shape constraint framework in the three imaging channels, determining the geometric reference of the visible light imaging image based on the spatial arrangement order of the corner points, and sequentially mapping the corner points of the infrared imaging image and the near-infrared imaging image to the geometric reference, and determining the geometric alignment result based on the turning direction of the continuous edge fold line.

[0058] Specifically, a marker shape constraint framework is deployed in the three imaging channels (independent data acquisition channels of the visible light imaging image, the infrared imaging image and the near-infrared imaging image). The marker shape constraint framework is a physical or virtual marker structure with a fixed geometric topology, which is located in the optical path between the optical entrance window and the image sensor, and is usually made of rigid materials with specific geometric shapes, such as a chessboard. The marker shape constraint framework can provide a reference for the detection of corner points. The spatial arrangement order of the corner points represents the key corner points in the image. For a chessboard marker, the corner points form a coordinate sequence in the order from left to right and from top to bottom. The geometric reference is determined based on the spatial arrangement order of the corner points of the visible light imaging image. The geometric reference represents the coordinate system with the visible light imaging image as the reference. The corner points of the infrared imaging image and the near-infrared imaging image are sequentially mapped to the geometric reference. The turning direction of the continuous edge fold line is constrained. The continuous edge fold line can be determined by the Canny edge detection algorithm, and the turning direction of the continuous edge fold line represents the direction change characteristics at the fold line inflection point, such as the turning angle when turning from a horizontal line segment to a vertical line segment. After mapping, it is verified whether the turning direction of the continuous edge fold line of the infrared imaging image and the near-infrared imaging image is consistent with the corresponding fold line of the visible light imaging image. If there is a deviation, the transformation parameters can be fine-tuned by thin plate spline interpolation, and then the geometric alignment result is output. By constraining the turning direction of the continuous edge fold line, the local deformation of the infrared imaging image and the near-infrared imaging image in the mapping process is limited, avoiding the risk of edge misplacement, distortion or breakage, and ensuring the reliability of multispectral image fusion.

[0059] In some embodiments of the present application, the light-dark extension directions of the high-reflection part, the low-reflection part, and the translucent tissue region are characterized according to the light-dark trend changes in the three imaging channels of the same scene, and the spectral corresponding structure is determined, including: performing light-dark trend analysis on the same field region of the three imaging channels, identifying the slow change section, the sudden change section, and the continuous growth section of the light-dark trend, determining the light-dark change structure, corresponding the boundary fold line position and the light-dark change structure, determining the light-dark transition of different spectral channels at the object boundary, and recording the light-dark extension directions of each spectral channel at the high-reflection part, the low-reflection part, and the translucent tissue region, and determining the spectral corresponding structure based on the light-dark extension directions.

[0060] Specifically, the same field region is also the physical space range observed by the visible light imaging image, the infrared imaging image, and the near-infrared imaging image, and the corresponding pixel region set of the corresponding physical space range in the three-channel image, and the light-dark trend change is also the trend of determining the luminance value gradient change of each pixel position in the visible light, infrared, and near-infrared wave bands, on the basis of which, the slow change section, the sudden change section, and the continuous growth section in the light-dark trend are identified. The slow change section indicates that the luminance value of a region in the image changes slowly along the pixel position, and the luminance difference of adjacent pixels is very small, and the whole presents a smooth and non-jumping form. The sudden change section indicates that the luminance value of a region in the image has a large change, rapidly transitions from a bright area to a dark area, or vice versa, forming an obvious luminance boundary. The continuous growth section indicates that the luminance value of a region in the image gradually increases in a single direction (such as horizontal, vertical, or inclined), without reverse jump, and presents a gradient rising form. The distribution characteristics and connection relationship of these sections can determine the light-dark change structure, that is, the overall framework describing the luminance change mode in the scene, including the alternating sequence of the slow change section and the sudden change section, the extension path of the continuous growth section, and the spatial topological relationship. The boundary fold line position and the light-dark change structure are corresponded, the boundary fold line position and the sudden change section in the light-dark change structure are accurately aligned through spatial coordinate matching, so as to determine the light-dark transition characteristics of different spectral channels at the object boundary. The boundary fold line position is generated when the corner point is detected and the contour is extracted. For the high-reflection part (such as a metal surface), the direction of the luminance value diffusion to the surrounding region is recorded, for the low-reflection part (such as a dark fabric), the path of the darkness value extension is recorded, and for the translucent tissue region (such as a translucent ceramic surface), the luminance gradient direction caused by the light transmission characteristics is recorded. Based on all the recorded information of the light-dark extension directions, the spectral corresponding structure is determined, that is, the framework of the mapping relationship of the luminance change directions under different spectral wave bands is established, so that the characteristics of the luminance change in the scene are accurately captured, and the consistency of the light-dark transition characteristics of the object boundary in the multi-spectral channels is ensured.

[0061] In some embodiments of the present application, when the geometric alignment result is regionally divided based on the spectral corresponding structure, the contour enhancement region, the texture detail region and the background smoothing region are determined, including: performing pixel neighborhood traversal on the geometric alignment result, extracting common boundary line segments constrained by continuous edge break lines, determining the pixel number ratio of the common boundary line segments in the pixel neighborhood, and determining the turning direction deviation of the common boundary line segments in the pixel neighborhood; when the pixel number ratio is greater than or equal to the pixel number ratio threshold value, and the turning direction deviation is less than or equal to the turning direction deviation threshold value, the region where the pixel neighborhood is located is determined as the contour enhancement region; extracting the junction line of the slowly varying segment and the continuously increasing segment from the spectral corresponding structure as the texture junction line, determining the texture density of the texture junction line in the pixel neighborhood, and determining the angle between adjacent texture junction lines as the junction trend deviation; when the texture density is greater than or equal to the texture density threshold value, and the junction trend deviation is less than or equal to the junction trend deviation threshold value, the region where the pixel neighborhood is located is determined as the texture detail region; determining the brightness gradient of the geometric alignment result in the pixel neighborhood, determining the maximum brightness gradient as the background gradient, and determining the brightness standard deviation of the geometric alignment result in the pixel neighborhood based on the spectral corresponding structure; when the background gradient is less than or equal to the background gradient threshold value, and the brightness standard deviation is less than or equal to the brightness standard deviation threshold value, the region where the pixel neighborhood is located is determined as the background smoothing region.

[0062] Specifically, pixel neighborhood traversal is performed on the geometric alignment result, the pixel neighborhood is a local window area centered on each pixel, usually a 5x5 pixel rectangular area, the entire geometric alignment result is scanned pixel by pixel, so that the region judgment is performed on each pixel neighborhood. In the judgment of the contour enhancement area, the common boundary line segment satisfying the continuous edge polyline constraint is extracted, the common boundary line segment is a boundary line segment existing in the visible light imaging image, the infrared imaging image and the near-infrared imaging image and satisfying the continuous edge polyline constraint, the proportion of the number of pixels covered by the common boundary line segment in the total number of pixels in the pixel neighborhood is calculated, that is, the proportion of the number of pixels covered by the common boundary line segment in the total number of pixels in the pixel neighborhood, and the turning direction deviation of the common boundary line segment in the pixel neighborhood (the angle between adjacent line segments in the common boundary line segment in the pixel neighborhood, reflecting the consistency of the boundary direction) is calculated. The pixel number proportion threshold is set to 30%, the pixel number proportion threshold is determined based on the boundary feature statistics of the multispectral image, when it is lower than 30%, the common boundary line segment in the pixel neighborhood accounts for too low a proportion, which may be scattered pixels caused by noise or false boundary, and cannot form an effective contour structure, and the turning direction deviation threshold is set to 15°. For each pixel neighborhood, the texture boundary line is extracted from the spectrum corresponding structure, the texture density of the texture boundary line in the current pixel neighborhood is calculated, that is, the ratio of the number of pixels belonging to the texture boundary line in the pixel neighborhood to the total number of pixels in the pixel neighborhood, and the angle between adjacent texture boundary lines is determined as the boundary direction deviation (reflecting the stability of the texture direction). The texture density threshold is set to 20%, when it is lower than 20%, the texture boundary line in the pixel neighborhood is too sparse to form an effective texture detail structure, the boundary direction deviation threshold is set to 30°, the brightness gradient of the geometric alignment result in the pixel neighborhood, that is, the brightness gradient of the visible light imaging image and the spatially mapped infrared imaging image and near-infrared imaging image, the maximum brightness gradient is determined as the background gradient, the brightness standard deviation refers to the dispersion degree of the brightness values of the visible light imaging image and the spatially mapped infrared imaging image and near-infrared imaging image in the pixel neighborhood based on the spectrum corresponding structure, reflecting the consistency of the brightness response of different spectral channels to the same neighborhood, and the background gradient threshold is set to 10%, when it is lower than 10%, the brightness in the pixel neighborhood changes gently, indicating a smooth and non-jumping feature, and when it is higher than 10%, the brightness changes relatively sharply, which may belong to the contour enhancement area or the texture detail area.The brightness standard deviation threshold is set to 10%, and when the brightness standard deviation exceeds 0.01, the brightness perception difference of different spectral channels to the same neighborhood is too large, the spectral stability is poor, and a uniform background smooth area cannot be formed. Through the dual constraints of the pixel number proportion threshold and the turning direction deviation threshold, the contour enhancement area is accurately determined to exclude edge breakage and geometric distortion interference. The synergistic effect of the texture density threshold and the boundary trend deviation threshold ensures the division of the texture detail area, which is based on the physical characteristics of the texture to ensure the retention of structural details. The background gradient threshold and the brightness standard deviation threshold determine the uniform background area, eliminate the interference of background noise on the fusion result, and improve the integrity of the fusion result.

[0063] In some embodiments of the present application, when the spectral corresponding structure is used to divide the geometric alignment result into regions, the contour enhancement area, the texture detail area and the background smooth area are determined, and the method further comprises: when a pixel neighborhood meets multiple region conditions at the same time, the region attribution is determined according to the priority of the contour enhancement area, the texture detail area and the background smooth area; when a pixel neighborhood does not meet all region conditions, the pixel neighborhood is taken as the center, the adjacent pixel neighborhoods which have been determined as the contour enhancement area, the texture detail area or the background smooth area are taken as adjacent pixel neighborhoods, the number of adjacent pixel neighborhoods belonging to the contour enhancement area, the texture detail area and the background smooth area is counted respectively, and the region to which the maximum number belongs is determined as the region attribution of the pixel neighborhood; when there are multiple maximum numbers, the region attribution is determined according to the priority of the contour enhancement area, the texture detail area and the background smooth area.

[0064] Specifically, when the geometric alignment result is regionally divided based on the spectral corresponding structure, the priority order is that the contour enhancement region is higher than the texture detail region, and the texture detail region is higher than the background smooth region. When a pixel neighborhood meets the determination conditions of the contour enhancement region, the texture detail region and the background smooth region at the same time, the pixel neighborhood is directly attributed to the contour enhancement region with the highest priority according to the priority order, and other conditions are ignored. The contour enhancement region is determined according to the common boundary line segment of the continuous edge polyline constraint, which has uniqueness (the contour of each object is unique) and irreplaceability (texture or background cannot replace the contour to define the object boundary). The texture intersection line of the texture detail region and the brightness gradient and the brightness standard deviation of the background smooth region are non-core features relative to the contour. The texture can exist in multiple directions and various forms, and the background can be smooth through different brightness levels. Both have certain replaceability, so the contour enhancement region is determined first. For the pixel neighborhood that does not meet all the region conditions, the adjacent pixel neighborhoods (usually including the 8 adjacent pixel neighborhoods in the up, down, left, right and diagonal directions) are defined with the pixel neighborhood as the center. The number of adjacent pixel neighborhoods that have been determined to belong to the contour enhancement region, the texture detail region or the background smooth region is counted. The region type with the largest number is determined as the region attribution of the pixel neighborhood. If there are multiple region types with the same number (such as the number of contour enhancement regions and texture detail regions is equal), the determination is still based on the priority order. The integrity of the object boundary in the fusion result is improved, the continuity of the texture detail region is retained, and the reliability of the fusion process is ensured.

[0065] In some embodiments of the present application, when the dominant contour line in the spatially mapped infrared imaging image is determined based on the boundary direction of the contour enhancement region, and the geometric edge in the visible light imaging image is corrected for direction consistency based on the dominant contour line, it includes: taking the boundary direction of the contour enhancement region as the reference, taking the high-contrast boundary in the spatially mapped infrared imaging image as the dominant contour line, determining the geometric edge contour line in the visible light imaging image, and adjusting the turning direction of the geometric edge contour line when the direction of the dominant contour line and the geometric edge contour line deviates. The dominant contour line and the adjusted geometric edge contour line are superimposed in the same direction.

[0066] Specifically, the boundary direction of the contour enhancement region is the local trend characteristic of the boundary line, and the high-contrast boundary in the spatially mapped infrared imaging image is taken as the dominant contour line. Since the imaging principle of the infrared imaging image is to capture the infrared radiation or reflection characteristics of the object, and different objects (or different parts of the same object) have essential differences in the absorption and reflection of infrared light due to differences in material, temperature, and density. These differences will directly lead to a clear and natural division in the brightness of the corresponding area in the infrared image, thereby forming a high-contrast boundary. For example, the temperature difference between the human body and the surrounding environment, the infrared reflection difference between metal materials and non-metal materials, the infrared penetration difference between translucent tissues and dense tissues, etc. will all form a clear brightness division in the infrared image. Therefore, these clear brightness divisions are determined as high-contrast boundaries, and they are taken as the dominant contour line. At the same time, the geometric edge contour line in the visible light imaging image is determined: the geometric edge contour line is a continuous boundary line formed by the geometric edges extracted from the visible light imaging image, which directly reflects the physical shape contour of the object in the visible light band. When the directions of the dominant contour line and the geometric edge contour line deviate, the turning direction of the geometric edge contour line is adjusted, and the turning angle of the inflection point of the geometric edge contour line is modified to match the turning direction of the dominant contour line. For example, if the dominant contour line turns 90 degrees to the left at the upper-left inflection point of the door frame, the corresponding inflection point turning angle of the geometric edge contour line is forced to correct to 90 degrees to the left. After adjustment, the dominant contour line and the adjusted geometric edge contour line are superimposed in the same direction to generate a contour fusion result, ensuring that the boundary directions are completely aligned, thereby improving the reliability of image fusion.

[0067] In some embodiments of the present application, when the texture lines in the spatially mapped near-infrared imaging image are superimposed in the direction of the texture direction based on the texture detail area, it includes: determining a texture framework based on the texture trend of all texture detail areas in the visible light imaging image, and taking the texture framework as a reference texture direction; along the reference texture direction, the texture lines in the spatially mapped near-infrared imaging image are superimposed into the texture framework; when the extension direction of the texture lines conflicts with the texture framework, the direction of the texture lines is converted, and the result of the direction conversion is superimposed into the texture framework; when the extension direction of the texture lines does not conflict with the texture framework, the extension direction of the texture lines is retained.

[0068] Specifically, the texture framework is a framework of global texture direction covering all texture detail areas in the entire visible light imaging image, which is determined according to the Gabor filter by analyzing the texture direction of each pixel neighborhood in the texture detail area, that is, the extension direction of the surface texture in the texture detail area, and is formed by spatial aggregation. The texture framework is taken as the reference texture direction, and the texture lines in the spatially mapped near-infrared imaging image are superimposed into the texture framework. Since the pixel coordinates of the spatially mapped near-infrared imaging image match the visible light imaging image, the texture lines represent the continuous line segments that can characterize the surface microstructure extracted by an algorithm such as edge detection (such as the Canny algorithm) of the spatially mapped near-infrared imaging image, and the superimposition process requires that the direction of the texture lines be coordinated with the reference texture direction, that is, the deviation of the extension direction of the texture lines (that is, the extension path direction of the texture lines in space) from the reference texture direction at the corresponding position of the texture framework is checked pixel by pixel, when the extension direction of the texture lines conflicts with the texture framework, the direction of the texture lines is converted, the direction conversion adjusts the direction of the texture lines to match the reference texture direction, for example, the texture lines inclined by 15 degrees are rotated to be parallel to the reference texture direction, or the included angle between them is less than or equal to the error allowed range. The error allowed range is determined according to the specific application scene of the multispectral fusion imaging camera module. In the medical observation scene (such as soft tissue texture observation), the error allowed range is relatively small to avoid misjudgment of the lesion texture caused by direction deviation, and in the security monitoring scene, the overall coherence of the texture needs to be considered, so the error allowed range can be appropriately relaxed to improve the fusion efficiency under the premise of ensuring texture recognition, thereby adapting to the demand of real-time monitoring. When the extension direction of the texture lines does not conflict with the texture framework, that is, the texture lines are parallel to the reference texture direction, or the included angle between them is less than or equal to the error allowed range, the extension direction of the texture lines is retained, and the texture lines are directly superimposed into the texture framework, which takes advantage of the high spatial resolution texture of the visible light imaging image, and at the same time fuses the special response of the near-infrared imaging image to the semi-transparent material, thereby improving the fusion quality of the texture detail area.

[0069] In some embodiments of the present application, when the brightness level of the visible light imaging image is superimposed based on the brightness extension direction of the background smooth area, it includes: determining the brightness extension direction of the background smooth area, and taking the brightness of the spatially mapped infrared imaging image as the dominant channel, taking the brightness extension direction as the brightness reference of the background smooth area, and superimposing the brightness reference based on the visible light imaging image.

[0070] Specifically, the brightness extension direction represents the gradual extension trend of the brightness value in the background smooth area along the spatial path, for example, in the sky background of security monitoring, it is manifested as the brightness gradient direction from top to bottom. The brightness of the spatially mapped infrared imaging image is taken as the dominant channel, the spatially mapped infrared imaging image has higher brightness stability in the background area, the uniform response of the infrared band to thermal radiation can effectively suppress the common illumination mutation noise in the visible light band, such as shadow or reflection interference, the brightness value sequence in the background smooth area of the spatially mapped infrared imaging image is extracted as the dominant brightness data, that is, in the spatial path of the background smooth area, the continuously sampled pixel brightness values are arranged in a set in spatial order, the brightness extension direction is taken as the brightness reference of the background smooth area, and the dominant brightness data of the dominant channel is projected to the brightness extension direction for piecewise superposition. The piecewise superposition divides the background smooth area into continuous small segments along the brightness extension direction, and the length of each segment is dynamically adjusted according to the complexity of the scene, and is usually 5-10 pixels. For each segment, the extracted dominant brightness data is fused according to the smoothing characteristics of the brightness reference. By taking the brightness of the spatially mapped infrared imaging image as the dominant channel, the uniformity advantage of the infrared band in the background smooth area is fully utilized, for example, in security monitoring, the local overexposure of the visible light image caused by night vehicle lights is eliminated, and it is ensured that the background smooth processing follows the natural brightness transition law of the scene, which not only retains the brightness level of the visible light imaging image, but also suppresses the noise interference through the dominant channel, and improves the reliability of the fusion process.

[0071] In some embodiments of the present application, when the connectivity of the boundary direction, the texture direction and the brightness extension direction is checked, when any area has a fracture, the boundary direction, the texture direction or the brightness extension direction is re-determined according to the fracture position and backtracking fusion, including: the connectivity of the boundary fold line of the contour enhancement area is detected, if the boundary fold line has a fracture, the turning direction of the boundary fold line at the fracture position is re-determined, and the direction consistency correction is re-performed, the continuity of the texture direction of the texture detail area is detected, if the texture framework has a sudden change in the texture direction, the texture direction of the texture framework with the sudden change in the texture direction is converted, and the extension superposition is re-performed, and the smoothness of the brightness extension direction of the background smooth area is detected, if the brightness extension direction has a fracture, the brightness extension direction is re-determined according to the position of the fracture point, and the superposition is re-performed.

[0072] Specifically, the connectivity of the boundary broken line of the contour enhanced region is detected: the boundary broken line is the boundary path composed of continuous edge broken lines in the contour enhanced region, and the connectivity detection is to analyze the topological continuity of the boundary broken line, check whether the end points of adjacent line segments are separated, if there is end point separation, that is, the boundary broken line is broken, the turning direction of the boundary broken line at the broken part is re-determined, and the direction consistency correction is re-performed, that is, the correction operation of the geometric edge contour line based on the dominant contour line, so that the boundary direction of the broken area is connected with the overall contour. The continuity of the texture direction of the texture detail region is detected, if the direction vectors of adjacent texture directions at the junction are large, that is, the texture is broken, such as the texture distortion caused by local noise, it is determined that the direction of the texture direction is suddenly changed, when the texture framework has a sudden change in the direction of the texture, the direction of the texture direction of the texture framework is converted, the texture direction of the sudden change region is adjusted by a thin plate spline interpolation algorithm to make it smooth transition, and the extendability is re-stacked, so that the texture detail is naturally continued on the sudden change region. The smoothness of the brightness extension direction of the background smooth region is detected, if the brightness change rate of adjacent pixel points in the brightness extension direction is large, that is, the brightness jump is not natural, such as the background spot caused by sensor noise, it is determined that the brightness extension direction is broken, then the smooth transition of the brightness extension path is generated by the brightness gradient field of the unbroken area on both sides of the broken point using a spline interpolation algorithm, and is re-stacked. If the connectivity check passes, that is, the boundary broken line is not broken, the texture framework does not have a sudden change in the direction, and the brightness extension direction is not broken, then the fusion image is output according to the contour fusion result, the texture fusion result and the background fusion result. By identifying the boundary break of the contour enhanced region, the sudden change of the texture direction of the texture detail region and the break of the background smooth region, the fusion quality is guaranteed.

[0073] The above embodiment synchronously constrains the positions and edge directions of the infrared imaging image and the near-infrared imaging image from the global and local levels by collecting visible light imaging images, infrared imaging images and near-infrared imaging images of three imaging channels, taking the visible light imaging image as a geometric reference, combining the spatial arrangement order of the corner points and the direction of the continuous edge fold line, avoiding the risk of local image distortion and edge misplacement caused by only simple registration, representing the light and dark extension direction based on the light and dark trend change of the same scene, determining the spectral corresponding structure and dividing the contour enhancement area, the texture detail area and the background smooth area, so that the fusion strategy of each area can adapt to its own characteristic attributes. The boundary direction of the contour enhancement area determines the dominant contour line of the infrared imaging image, corrects the geometric edge of the visible light imaging image, and ensures the contour clarity of the fusion process. The texture direction of the texture detail area superimposes the texture lines of the near-infrared imaging image, thereby enriching the details of the texture level. Relying on the brightness extension direction of the background smooth area, the smoothness of the background and the hierarchy during fusion are taken into account, thereby ensuring the continuity of the fused image. Through the connectivity check of the boundary direction, the texture direction and the brightness extension direction and the backtracking fusion, the risk of false images and discontinuity in the fused image can be avoided, and the reliability of the fusion result is improved.

[0074] Referring to Figure 2 As shown in the drawings, the embodiment provides a multispectral fusion imaging camera module for applying a multispectral fusion imaging camera module image processing method, which comprises a camera lens 1, a camera block 2 and a camera body 3. The camera lens 1 is used to shoot visible light imaging images, infrared imaging images and near-infrared imaging images. One end of the camera block 2 is connected to the camera lens 1, and the other end of the camera block 2 is connected to the camera body 3.

[0075] Specifically, the camera lens 1 can shoot visible light, infrared and near-infrared imaging images, which not only reduces the overall size of the module to adapt to installation scenes such as security equipment and portable medical imaging devices, but also ensures the field of view consistency of the three imaging channels. The camera block 2, as an intermediate connecting component of the camera lens 1 and the camera body 3, can stabilize and fix the optical attitude of the camera lens 1, reducing optical deviation during image acquisition. The modular structure of the camera lens 1, the camera block 2 and the camera body 3 reduces the maintenance cost during shooting operation.

[0076] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instruction"). It should also be noted that each block of the flowchart and / or block diagrams and a combination of flows and / or blocks in the flowchart and / or block diagrams can be implemented by computer program instructions. Although the computer program instructions can be implemented by hardware, the embodiments of the present application are not limited to a particular software configuration and / or hardware configuration. The computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instruction"). It should also be noted that each block of the flowchart and / or block diagrams and a combination of flows and / or blocks in the flowchart and / or block diagrams can be implemented by computer program instructions. Although the computer program instructions can be implemented by hardware, the embodiments of the present application are not limited to a particular software configuration and / or hardware configuration. The computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0077] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting the same. Although the present application is described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the

Claims

1. An image processing method of a multi-spectrum fusion imaging camera module, characterized in that, The method comprises the following steps: Collecting visible light imaging images, infrared imaging images and near-infrared imaging images of three imaging channels, taking the visible light imaging images as a geometric reference, performing spatial mapping on the infrared imaging images and the near-infrared imaging images based on the spatial arrangement order of the corner points and the direction of the continuous edge broken line, and determining a geometric alignment result; According to the light and dark trend changes of the same scene in the three imaging channels, the light and dark extension directions of the high-reflection part, the low-reflection part and the semi-transparent tissue region are characterized, the spectral corresponding structure is determined, the geometric alignment result is divided into regions based on the spectral corresponding structure, the contour enhancement region, the texture detail region and the background smoothing region are determined; Based on the boundary direction of the contour enhancement region, the dominant contour line in the spatially mapped infrared imaging image is determined, the geometric edge in the visible light imaging image is corrected based on the direction consistency of the dominant contour line, the contour fusion result is determined, the texture lines in the spatially mapped near-infrared imaging image are extended based on the texture direction of the texture detail region, and the texture fusion result is determined, and the brightness level of the visible light imaging image is superimposed based on the brightness extension direction of the background smoothing region, and the background fusion result is determined; The connectivity of the boundary direction, the texture direction and the brightness extension direction is checked, when any region is broken, the boundary direction, the texture direction or the brightness extension direction is re-determined according to the broken position and backtracking fusion is performed, and when the connectivity check is passed, the fused image is output.

2. The image processing method of the multispectral fusion imaging camera module according to claim 1, characterized in that, When the visible light imaging images are taken as the geometric reference, the infrared imaging images and the near-infrared imaging images are spatially mapped based on the spatial arrangement order of the corner points and the direction of the continuous edge broken line, and the geometric alignment result is determined, comprising: The spatial arrangement order of the corner points is determined based on a marker shape constraint framework in the three imaging channels, the geometric reference of the visible light imaging image is determined based on the spatial arrangement order of the corner points, the corner points of the infrared imaging image and the near-infrared imaging image are sequentially mapped to the geometric reference, and the turning direction of the continuous edge broken line is constrained to determine the geometric alignment result.

3. The image processing method of claim 2, wherein, When the light and dark trend changes of the same scene in the three imaging channels are characterized, the light and dark extension directions of the high-reflection part, the low-reflection part and the semi-transparent tissue region are characterized, and the spectral corresponding structure is determined, comprising: The light and dark trend of the same field of view region of the three imaging channels is analyzed, the slow change section, the sudden change section and the continuous growth section of the light and dark trend are identified, and the light and dark change structure is determined; The boundary broken line position and the light and dark change structure are corresponded to determine the light and dark transition of different spectral channels at the object boundary, and the light and dark extension directions of each spectral channel at the high-reflection part, the low-reflection part and the semi-transparent tissue region are recorded, and the spectral corresponding structure is determined based on the light and dark extension directions.

4. The image processing method of claim 3, wherein, When the geometric alignment result is divided into regions based on the spectral corresponding structure, the contour enhancement region, the texture detail region and the background smoothing region are determined, comprising: The pixel neighborhood of the geometric alignment result is traversed; extracting a common boundary line segment of a continuous edge polyline, determining a pixel quantity proportion of the common boundary line segment in a pixel neighborhood, and determining a turning direction deviation of the common boundary line segment in the pixel neighborhood, when the pixel quantity proportion is greater than or equal to a pixel quantity proportion threshold value and the turning direction deviation is less than or equal to a turning direction deviation threshold value, then determining a region in which the pixel neighborhood is located as the contour enhancement region; extracting a junction line of a slowly-varying segment and a continuously-increasing segment from the spectrum corresponding structure as a texture junction line, determining a texture density of the texture junction line in a pixel neighborhood, and determining an intersection strike deviation of adjacent texture junction lines, when the texture density is greater than or equal to a texture density threshold value and the intersection strike deviation is less than or equal to an intersection strike deviation threshold value, then determining a region in which the pixel neighborhood is located as the texture detail region; determining a luminance gradient of the geometric alignment result in the pixel neighborhood, determining a maximum luminance gradient as a background gradient, determining a luminance standard deviation of the geometric alignment result in the pixel neighborhood based on the spectrum corresponding structure, when the background gradient is less than or equal to a background gradient threshold value and the luminance standard deviation is less than or equal to a luminance standard deviation threshold value, then determining a region in which the pixel neighborhood is located as the background smoothing region.

5. The image processing method of claim 4, wherein, In the region division of the geometric alignment result based on the spectrum corresponding structure, the contour enhancement region, the texture detail region and the background smoothing region are determined, and the method further comprises: when the pixel neighborhood meets multiple region conditions at the same time, then determining the region attribution according to the priority of the contour enhancement region, the texture detail region and the background smoothing region; when the pixel neighborhood does not meet all the region conditions, then taking the pixel neighborhood as the center, taking adjacent pixel neighborhoods which have been determined as the contour enhancement region, the texture detail region or the background smoothing region as adjacent pixel neighborhoods, respectively counting the number of the contour enhancement region, the texture detail region and the background smoothing region in the adjacent pixel neighborhoods, and determining the region attribution of the pixel neighborhood according to the region to which the maximum number belongs, when the maximum number is the same, then determining the region attribution according to the priority of the contour enhancement region, the texture detail region and the background smoothing region.

6. The image processing method of claim 5, wherein, In the determination of the dominant contour line in the spatially mapped infrared imaging image based on the boundary direction of the contour enhancement region, and the direction consistency correction of the geometric edge in the visible light imaging image based on the dominant contour line, the method comprises: taking the boundary direction of the contour enhancement region as the reference, taking a high-contrast boundary in the spatially mapped infrared imaging image as the dominant contour line, and determining a geometric edge contour line in the visible light imaging image; when there is a deviation between the directions of the dominant contour line and the geometric edge contour line, then adjusting the turning direction of the geometric edge contour line, and superimposing the dominant contour line and the adjusted geometric edge contour line in the same direction.

7. The image processing method of claim 6, wherein, In the extension superimposition of the texture line in the spatially mapped near-infrared imaging image based on the texture direction of the texture detail region, the method comprises: determining a texture framework based on the texture strike of all the texture detail regions in the visible light imaging image, and taking the texture framework as the reference texture direction; superimposing the texture lines in the spatially mapped near-infrared imaging image into the texture framework along the reference texture direction; when the extension direction of the texture lines conflicts with the texture framework, the direction of the texture lines is converted, and the converted result is superimposed into the texture framework; when the extension direction of the texture lines does not conflict with the texture framework, the extension direction of the texture lines is retained.

8. The image processing method of claim 7, wherein, when superimposing the brightness gradation of the visible light imaging image based on the brightness extension direction of the background smooth area, comprising: determining the brightness extension direction of the background smooth area, taking the brightness of the spatially mapped infrared imaging image as the dominant channel, taking the brightness extension direction as the brightness reference of the background smooth area, and superimposing the brightness reference based on the visible light imaging image.

9. The image processing method of claim 8, wherein, when the connectivity of the boundary direction, the texture direction and the brightness extension direction is checked, when any area has a fracture, the boundary direction, the texture direction or the brightness extension direction is re-determined according to the fracture position and backtracking fusion, comprising: the connectivity of the boundary fold line of the contour enhancement area is detected, if the boundary fold line has a fracture, the turning direction of the boundary fold line at the fracture is re-determined, and the direction consistency correction is re-performed; the continuity of the texture direction of the texture detail area is detected, if the texture framework has a sudden change in the texture direction, the texture direction of the texture framework with the sudden change in the texture direction is converted, and the extension superposition is re-performed; the smoothness of the brightness extension direction of the background smooth area is detected, if the brightness extension direction has a fracture, the brightness extension direction is re-determined according to the position of the fracture, and the superposition is re-performed.

10. A multispectral fusion imaging camera module for applying the image processing method of any one of claims 1-9, wherein comprising: a camera lens, a camera block and a camera body; the camera lens is used for shooting visible light imaging image, infrared imaging image and near-infrared imaging image; one end of the camera block is connected with the camera lens, and the other end of the camera block is connected with the camera body.

Images