Infrared and visible light video mimicry fusion method based on difference feature correlation synthesis and intuitive possible set

Abstract

The invention relates to a multi-mode video fusion method, in particular to an infrared and visible light video mimicry fusion method based on difference feature correlation synthesis and an intuitive possible set, and the method comprises the steps: constructing a difference feature set, and recognizing the significant difference feature of each frame of image; constructing a feature incidence matrix through correlation coefficients, selecting corresponding operators according to element values of the incidence matrix to carry out distribution synthesis, and establishing difference feature saliency correlation synthesis based on multi-rule combination; establishing an intuitive possible set division criterion, and defining a high-fusion effective interval as a significant fusion information area; and realizing self-adaptive optimization of the fusion algorithm by combining the regional dominance and the average fusion validity. Compared with a mainstream fusion algorithm, the method has the advantages that infrared typical target and visible light background details can be kept in a more balanced manner on the subjective vision level; on the objective evaluation level, multiple indexes such as edge strength, mutual information and spatial frequency are all remarkably improved, and effectiveness and superiority in the aspect of improving fusion quality and system adaptive capacity are verified.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-modal video fusion method, in particular to an infrared and visible light video morphing fusion method based on difference feature correlation and intuitive possibility set. BACKGROUND

[0002] Infrared and visible light video fusion technology has important significance in improving imaging quality and information acquisition capability. Infrared video is based on the thermal radiation difference between target and background, which can realize effective detection in complex environments such as weak light, foggy weather and camouflage, and has all-weather working ability and strong anti-interference ability, but the imaging results often lack detailed information and have poor visibility; visible light video is based on light reflection characteristics, which can provide rich scene texture and spatial details. Effective fusion of these two complementary video modalities not only can integrate their respective feature advantages, but also can significantly improve the imaging quality and accuracy of the detection system, and thus has wide application value in the fields of unmanned aerial vehicle remote sensing observation, intelligent driving, air defense guidance and water conservancy monitoring.

[0003] In recent years, fusion methods based on difference feature driving have gradually become an important research direction to improve image fusion quality. Existing research shows that making full use of the complementary information between different modal images can effectively enhance the visual quality and information integrity of the fusion results. For example, Yan et al. proposed a hierarchical fusion strategy for the differences between infrared and visible light images in background and target performance, respectively processing the bottom information and detailed features, which significantly improved the complementary advantages of the fusion image; Jiao et al. designed a fusion scheme based on Laplacian pyramid by extracting significant contour and contrast features in medical image fusion, which achieved obvious improvement in structural similarity and peak signal-to-noise ratio; Sappa et al. proposed a fusion idea of dynamically adjusting wavelet type according to features by analyzing the matching relationship between image features and wavelet bases. These studies show that the effective representation and quantitative analysis of difference features play a key role in the selection and strategy optimization of fusion algorithms.

[0004] However, the existing method still has obvious deficiencies in the multi-attribute coordination and correlation modeling of heterogeneous difference characteristics. Different difference characteristics often have different dimensions and orders of magnitude. If only simple standardization is performed without considering the distribution characteristics of the feature amplitude in the scene, the distribution estimation deviation is easy to occur, which further affects the fusion effect and even leads to the failure of the method. In addition, the existing model mainly focuses on the influence of single attribute or single class difference characteristics on fusion, and does not fully consider the dynamic changes and correlation between multiple attributes, resulting in a lack of semantic interpretability in the fusion process and limited performance improvement. Therefore, in-depth research on the influence of the type, amplitude, distribution and internal correlation of difference characteristics on the fusion result, and the establishment of the deep set mapping relationship between multi-attribute and fusion algorithm, have become key problems to be solved at present. SUMMARY

[0005] The present application is aimed at the problem that the existing fusion model is difficult to adapt to the dynamic changes of multi-attribute and correlation of difference characteristics, and proposes an infrared and visible light video quasi-state fusion method based on difference characteristic correlation combination and intuitionistic fuzzy set.

[0006] The present application is implemented by adopting the following technical scheme: the infrared and visible light video quasi-state fusion method based on difference characteristic correlation combination and intuitionistic fuzzy set first constructs a difference characteristic set to quantitatively represent the essential difference between infrared and visible light videos in radiation characteristics, structural texture and scene content; the K nearest neighbor non-parametric estimation method is used to analyze the frequency distribution of the feature amplitude, and the difference characteristic saliency is constructed by fusing the amplitude and frequency information to coordinate the competition and complementary relationship between multiple attributes and objectively identify the salient difference characteristics of each image; further, the Pearson correlation coefficient is used to construct a feature correlation matrix to explicitly represent the internal correlation between heterogeneous difference characteristics; according to the element values of the obtained correlation matrix, the corresponding T operator is selected to perform distribution synthesis, and the difference characteristic correlation combination value corresponding to the feature correlation combination is calculated, so as to establish the difference characteristic saliency correlation combination based on multi-rule combination; according to the fusion result of the difference characteristic amplitude, the intuitionistic fuzzy set theory is introduced to construct a two-dimensional quantitative mechanism of fusion effectiveness and information reliability, and the salient fusion information area with high fusion effectiveness and high reliability is accurately divided; finally, the fusion score index function is constructed, and the uncertainty reasoning result provided by the regional dominance rate, the average fusion effectiveness and the intuitionistic fuzzy set is combined to realize the adaptive optimization of the fusion algorithm.

[0007] The above infrared and visible light video quasi-state fusion method based on difference characteristic correlation combination and intuitionistic fuzzy set has the following identification process of salient difference characteristics:

[0008] The difference characteristic saliency is defined as a dynamic function representing the relative importance of different attributes in the whole image, , is the first In-frame difference features Significance of difference features , Let be the left and right boundaries of the feature set, respectively. For the difference characteristic amplitude The probability density distribution;

[0009] A judgment criterion is constructed by introducing the golden ratio. In the formula, Represents video frames The significance of the difference feature in the video frame is used to determine whether the difference feature is significant. If the significance of the difference feature is greater than or equal to a threshold, then the difference feature is considered a video frame difference feature. Significant differences in characteristics.

[0010] The aforementioned infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possibility set constructs the feature association matrix as follows:

[0011] The feature correlation matrix is ​​constructed using the Pearson correlation coefficient. , In the formula, , Indicates differences in characteristics Significant differences were selected from the data. This represents a combination of features. Indicates significant differences Significant differences The Pearson correlation coefficient between them The total number of significantly different features;

[0012] Pearson correlation coefficient The calculation is as follows: ,in, Indicates significant differences The fusion effectiveness score, Indicates significant differences The fusion effectiveness score;

[0013] in, , In the formula, Indicates that variable A uses the fusion algorithm. k The result of the first Significant difference features of the frame-fused image in pixel block ξ amplitude, Indicates that variable A uses the fusion algorithm. k The result of the first Significant difference features of the frame-fused image in pixel block ξ amplitude, , respectively represent the significant difference feature frame infrared and visible light source image in pixel block ξ amplitude, , respectively represent the significant difference feature frame infrared and visible light source image in pixel block ξ amplitude; represent the similarity measure; , and , respectively represent the significant difference feature , the significant difference feature weight coefficient.

[0014] The above-mentioned infrared and visible light video quasi-state fusion method based on difference feature association and intuitionistic possibility set, the high fusion effective interval established is wherein, , .

[0015] The above-mentioned infrared and visible light video quasi-state fusion method based on difference feature association and intuitionistic possibility set, the adaptive optimization process of the fusion algorithm is:

[0016] The fusion algorithm argument A k all feature association combinations involved are , define

[0017] Area dominant rate , represent the frequency of the fusion algorithm argument A k in the significant fusion information area of the association combination , is the number of times that the difference feature association value of the fusion algorithm argument A k falls into the significant fusion information area in the association combination , is the total number of times that the difference feature association value of all fusion algorithm arguments falls into the significant fusion information area in the association combination ;

[0018] Define the average fusion effective degree , represent the average performance of the fusion algorithm argument A k in the significant fusion information area, is the difference feature association value of the fusion algorithm argument A k when falling into the significant fusion information area for the jth time;

[0019] For each association combination , calculate the fusion algorithm argument Ak Fusion score

[0020] After obtaining the comprehensive score of each fusion algorithm variable, the optimal fusion strategy is determined according to the maximization principle. ,in This represents the optimal fusion algorithm variable for the current frame.

[0021] The infrared and visible light video mimicry fusion method described above, based on differential feature association synthesis and intuitionistic possible set, includes six differential features in the differential feature set: grayscale mean, standard deviation, average gradient, edge intensity, roughness, and contrast.

[0022] The aforementioned infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possible sets, with differential feature amplitude... , , They represent the first Differences between mid-frame infrared video and visible light video The amplitude on.

[0023] This invention proposes a fusion method for infrared and visible light video mimicry based on the multi-mimicry mechanism of mimicking an octopus and using differential feature association synthesis and intuitionistic possibility sets. By constructing a set-valued mapping relationship between the salience of differential features and the variables of the fusion algorithm, the fusion strategy is adaptively adjusted to the multiple attributes of features and their associated structures, effectively overcoming the limitation in fusion performance caused by traditional models ignoring the intrinsic correlation between features. Experiments on the BEPM and TNO datasets show that this method has significant advantages in preserving infrared target features and visible light structural details, and the fusion quality outperforms existing single fusion methods in both subjective visual effects and multiple objective evaluation metrics. Attached Figure Description

[0024] Figure 1 This is a flowchart of the overall process of the method of the present invention.

[0025] Figure 2 This is a schematic diagram of infrared and visible light video frames.

[0026] Figure 3 This is a schematic diagram showing the significance values ​​of the difference features corresponding to the two sets of scenarios.

[0027] Figure 4 This is the composite image of the differential features associated with the first frame.

[0028] Figure 5 The image shows the result of associating differential features with the shadow in frame 1.

[0029] Figure 6 This is the composite image of the differential features in frame 120.

[0030] Figure 7 The difference feature correlation result for the 120th frame is associated with the falling shadow result image.

[0031] Figure 8 The difference feature correlation result for the 200th frame is associated with the integrated and associated falling shadow result image.

[0032] Figure 9 The contrast fusion results of the 1st frame and the 120th frame of the BEPM dataset are shown from left to right as follows: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, SWT, WPT, and the result of the method of the application.

[0033] Figure 10 The contrast fusion results of the 140th frame and the 165th frame of the BEPM dataset are shown from left to right as follows: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, SWT, WPT, and the result of the method of the application.

[0034] Figure 11 The contrast fusion results of the 185th frame and the 200th frame of the BEPM dataset are shown from left to right as follows: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, SWT, WPT, and the result of the method of the application.

[0035] Figure 12 The contrast fusion results of the TNO dataset are shown from left to right and from top to bottom as follows: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, SWT, WPT, and the result of the method of the application. DETAILED DESCRIPTION

[0036] The core of video fusion is to effectively integrate the difference features between different modal videos and give full play to the respective imaging advantages. Dynamic selection of fusion strategies according to the perceived difference features is the key to realize mimicry fusion, and therefore, effective characterization and quantitative description of the difference complementary information in multi-modal videos become an important prerequisite for the intelligent fusion system to realize accurate analysis and decision-making.

[0037] Current video fusion faces three main challenges: First, dissimilar features differ significantly in size and magnitude. Directly using the original feature values ​​for analysis would overemphasize the role of features with higher values ​​while weakening the contribution of features with lower values, leading to an imbalance in the importance of features. Second, in the same imaging scene, similar dissimilar features corresponding to the background and target areas often exhibit a non-uniform distribution. The spatial distribution density of their feature amplitudes directly affects the construction and evaluation of fusion effectiveness, thus impacting fusion performance. Furthermore, existing methods generally ignore the intrinsic relationships between dissimilar features, resulting in a lack of sufficient semantic interpretability in the fusion process, which restricts further improvement in fusion results.

[0038] To address the aforementioned problems, this invention proposes a method for fusing infrared and visible light video mimicry based on differential feature association synthesis and intuitionistic possibility sets. The overall process is as follows: Figure 1 As shown.

[0039] Quantitative description of differential characteristics and their attributes

[0040] Infrared and visible light sensors, based on different radiation physics characteristics and imaging mechanisms, detect scene targets from two dimensions: thermal radiation and light reflection, respectively, thus forming imaging information with significantly complementary characteristics. Figure 2 (As shown in the red box). Infrared video primarily reflects the thermal radiation characteristics of a target, offering a significant advantage in representing the outline of thermal targets; while visible light video provides richer information on scene structure and detail. The differences in imaging characteristics between the two types of video are mainly reflected in three aspects: brightness distribution, edge structure, and texture detail.

[0041] To effectively quantify the above differences, this invention selects six feature indicators for difference characterization: mean gray level (GM) is used to directly characterize the overall brightness level of the image; standard deviation (SD) reflects the dispersion of image gray levels, and the larger the value, the stronger the image contrast and the richer the effective information contained; average gradient (AG) reflects the image's ability to express minute details, and the higher the value, the better the image's visual clarity; edge intensity (EI) is extracted based on the Sobel operator and is used to describe the intensity distribution of contour and structural features in the image; roughness (CA) and contrast (CN) are used to characterize the roughness of the texture structure and the overall contrast characteristics of the pixel intensity distribution, respectively.

[0042] In summary, this invention uses GM to characterize brightness differences, SD, AG and EI to describe the contrast characteristics, sharpness and intensity distribution in edge information, and CA and CN to quantify texture features, thereby constructing a comprehensive feature set covering three key differences, providing a basis for the intelligent selection of subsequent fusion strategies.

[0043] Difference characteristic amplitude The absolute difference of the infrared and visible light video corresponding frames in the feature is quantified, and the calculation is shown as formula (1): wherein , respectively represent the amplitude of the infrared video and the visible light video in the difference feature of the first frame.

[0044] (1)

[0045] The difference feature frequency attribute reflects the distribution width of the feature in the imaging scene from a macro perspective, and reflects the local density degree of the feature with the amplitude change from a micro perspective. In order to obtain the distribution characteristics of the attribute, the K nearest neighbor non-parametric estimation method is used to model the probability density of the difference feature amplitude. In the specific implementation, first, the video sequence is divided into non-overlapping blocks frame by frame by using an m*n smoothing window, each type of difference feature is extracted, and an initial amplitude sample set is formed , each frame contains amplitude points. In view of the problem of limited number of samples, the step is set, the sample set is expanded to by interpolation, and the number of samples is increased to: , , for the left and right boundaries of the sample set respectively. For any sample in the expanded sample set , the probability density estimation value is given by formula (2):

[0046] (2)

[0047] wherein is twice the distance of the sample point to its nearest neighbor sample, , by point-by-point calculation, the probability density distribution of each difference feature amplitude can be obtained. Calculation of difference feature saliency and determination of significant difference feature

[0048] The difference feature saliency is defined as a dynamic function representing the relative importance of different attributes in the whole image, as shown in formula (3):

[0049]

[0050] (3)

[0051] is the difference feature in the first frame.The significance of the difference features is achieved by mapping the magnitude and frequency attributes of similar difference features to the synthesis axis, thus realizing a unified representation of multiple attributes of dissimilar difference features and simplifying the subsequent feature synthesis process.

[0052] In bimodal video sequences, the saliency of differences in brightness, edge, and texture information contained in each frame varies significantly, resulting in a hierarchy of importance among the features. Figure 3 Taking the two sets of infrared and visible light images as examples: (a) The scene includes targets such as people, billboards, utility poles, and vehicles, and the main difference lies in brightness and texture features; (b) The scene is mainly composed of vegetation, and the difference is concentrated in brightness and edge information. The significance values ​​of the difference features between the two sets of images ( Figure 3 This further validated the analysis results.

[0053] Therefore, this invention proposes the concept of "significant difference features" to describe feature categories that have more significant difference information compared to other features in multimodal images. Selecting such features to guide the subsequent mimicry fusion process has clear physical meaning and application value. The significance of difference features effectively coordinates the relationship between multiple attributes, and its value has good rationality and comprehensiveness. To accurately identify significant difference features in each video frame, this invention introduces the golden ratio to construct a judgment criterion, as shown in equation (4):

[0054] (4)

[0055] In the formula, Represents video frames The significance of the difference feature in the video frame is used to determine whether the difference feature is significant. If the significance of the difference feature is greater than or equal to a threshold, then the difference feature is considered a video frame difference feature. Significant differences in characteristics.

[0056] Construction of Feature Correlation Matrix and Correlation Composition

[0057] To quantify the intrinsic relationships between significantly different features, this invention uses the Pearson correlation coefficient to construct a feature correlation matrix. , (5)

[0058] In the formula, , Indicates differences in characteristics Significant differences were selected from the data. This represents a combination of features. Indicates significant differences Significant differences The correlation coefficient between them This represents the total number of significantly different features. The correlation coefficient... The calculation is shown in equation (6):

[0059] (6)

[0060] A value close to 1 or -1 indicates a strong linear correlation, while a value close to 0 indicates no linear correlation.

[0061] in, Indicates significant differences The fusion effectiveness score, Indicates significant differences The fusion effectiveness score;

[0062] (7)

[0063] (8)

[0064] In the formula, Indicates that variable A uses the fusion algorithm. k The result of the first Significant difference features of the frame-fused image in pixel block ξ amplitude, Indicates that variable A uses the fusion algorithm. k The result of the first Significant difference features of the frame-fused image in pixel block ξ amplitude, , They represent the first Significant differences in pixel block ξ between infrared and visible light source images. amplitude, , They represent the first Significant differences in pixel block ξ between infrared and visible light source images. The amplitude; It represents a similarity measure.

[0065] Determined dynamically by characteristic amplitude:

[0066] , (9)

[0067] , (10)

[0068] , and , They represent significant differences. Significant differences The weighting coefficients.

[0069] Possibility distribution synthesis can obtain more accurate information representation and prediction, which is to synthesize the distribution of multi-source information according to certain rules. The operators of distribution synthesis include ring product operator, T operator, intersection operator, union operator, average operator, S-module operator, etc., wherein the T operator (conjunction operator or minimum operator) is suitable for information processing in the case that various information has high overlap, and can effectively solve the problem of information redundancy.

[0070] The characteristic correlation matrix T and T operator based on the Pearson correlation coefficient are combined to obtain the specific calculation formula of the T operator, as shown in Table 1. That is, according to the obtained correlation matrix element value, the corresponding T operator is selected for distribution synthesis, and the difference characteristic correlation value Q corresponding to the characteristic correlation combination is calculated, so as to establish the difference characteristic saliency correlation based on multi-rule combination.

[0071] Table 1 Calculation rule of T operator

[0072] Variable optimization of fusion algorithm based on intuitionistic fuzzy set

[0073] In order to quantitatively analyze the ability of different fusion algorithms to maintain significant difference characteristics, the present application combines the intuitionistic fuzzy set theory and the cosine similarity measure to propose a fusion effectiveness evaluation model. The model realizes multi-level evaluation of algorithm performance by constructing the possibility distribution of fusion effect.

[0074] According to the fusion result of the difference characteristic amplitude, the present application establishes a three-interval intuitionistic fuzzy set division criterion:

[0075] 1) High fusion effective interval , which indicates that the fusion result is significantly better than the benchmark fusion method in feature preservation, and at this time, there is a clear possibility of high fusion effect set.

[0076] 2) Medium fusion effective interval , which indicates that the fusion effect is in a critical state, and the feature preservation ability is uncertain.

[0077] 3) Low fusion effective interval , which reflects that the fusion algorithm fails to effectively preserve the feature information of the source image, and there is a clear possibility of low fusion effect set.

[0078] Wherein, the threshold , respectively correspond to the lowest acceptable performance level and the optimal benchmark. , .

[0079] The high fusion effective interval is defined as the significant fusion information region, which represents the stable and excellent fusion performance of the fusion algorithm on the significant difference feature.

[0080] Let the fusion algorithm argument A k All feature association combinations involved are , and the performance indicators are defined as follows:

[0081] 1) Area dominance rate: the fusion algorithm argument A k The frequency of the significant fusion information region in the association combination reflects the fusion stability of the algorithm:

[0082] (11)

[0083] Where is the difference feature association value of the fusion algorithm argument A k in the association combination , is the total number of times that the difference feature association value of all fusion algorithm arguments in the association combination falls into the significant fusion information region.

[0084] 2) Average fusion effectiveness: the average performance of the fusion algorithm argument A k in the significant fusion information region, which is defined as follows:

[0085] (12)

[0086] Where is the difference feature association value of the fusion algorithm argument A k when it falls into the significant fusion information region for the jth time.

[0087] For each association combination , the fusion score of the fusion algorithm argument A k is calculated:

[0088] (13)

[0089] After obtaining the comprehensive score of each fusion algorithm argument, the optimal fusion strategy is determined according to the maximization principle:

[0090] (14)

[0091] Where The optimal fusion algorithm argument of the current frame is represented. The preferred mechanism realizes the scene content adaptive mimicry fusion by dynamically evaluating the performance of each algorithm on different difference characteristics.

[0092] The fusion algorithm argument constructed by the application aims to systematically investigate the performance of different transformation mechanisms in the multi-scale fusion framework. Seven representative multi-scale transformation algorithms are selected, which are denoted as A1 to A7 in turn, and specifically include: Curvelet transform based on anisotropic geometric feature representation (A1), dual-tree complex wavelet transform with good translation invariance and direction selectivity (A2), classical multi-scale and multi-resolution Laplacian pyramid transform (A3), and non-subsampled contourlet transform (A4) and non-subsampled shearlet transform (A5) which can provide more flexible direction decomposition, and static wavelet transform (A6) which is improved on traditional wavelet transform to avoid frequency aliasing, and wavelet packet transform (A7) which can realize more fine high-frequency subband decomposition.

[0093] Experimental verification

[0094] Experiments are conducted on two public data sets to demonstrate the advantages and robustness of the proposed method. The first data set BEPM (Bristol Eden Project Multi-sensor) involves a soldier walking to the camera, walking along a path away from the camera, and then crossing an empty field from left to right. The video sequence was taken with visible light and infrared at the Eden Project Biomes in Cornwall, UK. The application only selects the first 200 aligned frames as the research object, and the image size is 576*480.

[0095] The second data set is the TNO data set, which provides enhanced visual (390-700nm), near-infrared (700-1000nm) and long-wave infrared (8-12um) night images of different military and surveillance scenes, showing different objects and targets (such as people, vehicles) in different backgrounds (such as rural areas, cities). The data set will help develop static and dynamic image fusion algorithms, color fusion algorithms, multi-spectral target detection and recognition algorithms, and weak target detection algorithms. The Nato_camp_sequence containing 32 image pairs is selected for verification, and all images are 360*270 pixels.

[0096] The application selects seven algorithms in the multi-scale fusion framework, specifically including: Curvelet transform (CVT), dual-tree complex wavelet transform (DTCWT), Laplacian pyramid transform (LP), non-subsampled contourlet transform (NSCT), non-subsampled shearlet transform (NSST), static wavelet transform (SWT) and wavelet packet transform (WPT), denoted as A1-A 7。

[0097] Experimental cases

[0098] Take the selected infrared and visible videos of BEPM dataset as an example to illustrate the overall process of the method. First, the amplitude of each type of difference feature in each frame of the video sequence is calculated using formula (1). The m*n smoothing window is used to perform non-overlapping block processing on the infrared and visible video sequence frame by frame to extract the corresponding feature information. In order to balance the contradiction between the composite features introduced by too large blocks and the difficulty of extracting features by too small blocks, it is stipulated that m=n=16. According to formula (3), the amplitude and frequency attributes of each type of difference feature in each frame are calculated to obtain the corresponding difference feature saliency. On this basis, the saliency values of each feature are screened and sorted according to the feature discrimination condition, so as to determine the salient difference features in each video frame. Table 2 shows the analysis results of the 1st, 50th, 120th, 140th, and 200th frames. The first column is the video frame number, the second column is the discrimination threshold calculated according to formula 4, and the third to eighth columns are the saliency values of the six types of difference features (GM, SD, AG, EI, CN, and CA). The features marked in bold in the table are the salient difference features of the corresponding frame, which include GM, AG, and CA for the 1st frame, GM and CA for the 50th frame, AG, EI, and CA for the 120th frame, CA for the 140th frame, and AG and CA for the 200th frame.

[0099] Table 2 Salient difference features corresponding to each frame

[0100] Based on the Pearson correlation coefficient (formula 6), the correlation between each pair of the six types of difference features is calculated to construct the feature correlation matrix T corresponding to each frame. For ease of analysis, the correlation between features is displayed in table form. Table 3, Table 4, and Table 5 list the correlation matrix data of the 1st frame, the 120th frame, and the 200th frame, respectively. The rows and columns of the matrix correspond to the six types of difference features, and the features marked in bold are the salient difference features of the current frame, which are consistent with the results in Table 2. For subsequent joint integration, the correlation coefficients between the salient difference features are marked in bold font. Based on the element values in the correlation matrix, the appropriate T operator is selected for fusion rule synthesis. Taking the 1st frame in Table 3 as an example, the salient difference features are GM, AG, and CA. The correlation coefficients of GM and AG, and AG and CA are all close to 0, belonging to the unrelated relationship, so T3 operator is selected. GM and CA are negatively correlated, so T2 operator is selected. The selection method of other frames is similar.

[0101] Table 3 Feature correlation matrix of the 1st frame

[0102] Table 4 Feature correlation matrix of the 120th frame

[0103] Table 5 Feature correlation matrix of the 200th frame

[0104] Firstly, the fusion effectiveness of each difference feature under different algorithm arguments is calculated according to formula (5). Then, the correlation between each pair of features is determined according to the significant difference features and their corresponding T-operators. In the synthesis process, the algorithm argument with the maximum fusion effectiveness is selected by using the disjunction operator. Finally, the correlation shadow of the feature synthesis results under different algorithm arguments is constructed and mapped to the combination space of the corresponding difference feature significance. It should be noted that if there is only one significant difference feature in a frame, the steps of feature correlation matrix calculation and correlation shadow construction are skipped, and the algorithm argument is directly selected according to the fusion effectiveness of the feature; if there are multiple significant difference features (such as {fa, fb, fc}), the correlation between all feature pairs (such as <fa,fb>, <fa,fc>, <fb,fc>) needs to be considered comprehensively.

[0105] By counting the frequency of each algorithm argument in the significant fusion information area and its average fusion effectiveness, the fusion score index is calculated using a weighted method. The total score under each feature correlation combination (formula 14) is summarized to determine the optimal quasi-state fusion algorithm argument.

[0106] Figure 4 The correlation results of the three significant difference features (GM, AG, and CA) of the first frame are shown. From the figure, it can be found that the main fusion effective information of the image is mainly distributed in the part with small significant difference feature values in the correlation shadow diagram. Figure 6 With Figure 7 The correlation and correlation shadow results of the three significant difference features (AG, EI, CA) of the 120th frame are shown. Based on the fusion score sum in Table 7, algorithm A4 is selected as the optimal quasi-state argument for this frame. Figure 8 The feature correlation shadow of the 200th frame is shown. Since this frame only contains two significant difference features AG and CA, there is only one correlation combination. Combined with Table 8 and Figure 8 The results show that the fusion performance of algorithm argument A5 is significantly better than that of other arguments.

[0107] Table 6 Fusion score index values of different quasi-state arguments for the first frame

[0108] Table 7 Fusion score index values of different quasi-state arguments for the 120th frame

[0109] Table 8 Fusion score index values of different metamorphic variables of the 200th frame

[0110] Experimental result analysis

[0111] In order to verify the rationality and effectiveness of the method proposed in the present application, a comparison is made with selected multi-scale fusion algorithms, and the parameter settings of the fusion algorithms are set according to the original literature. At the same time, due to the subjective evaluation being easily affected by the personal psychological factors, mental state and other aspects of the evaluator, there is a certain subjective initiative, and 7 objective evaluation indexes are used to evaluate the fusion results of the algorithm in all aspects. Including, Q AB / F , Q0, Q w , Q e , mutual information (MI), visual information fidelity (VIFF) and structural similarity (SSIM). The higher the index value, the better the fusion performance. In the experiment, the optimal value and the suboptimal value are highlighted in bold.

[0112] Fusion results of the BEPM data set

[0113] First, some fusion results of several frames are analyzed, as shown in Figures 9-11. From the fusion results, it can be seen that a typical single fusion algorithm cannot maintain good fusion effect for all frames, and with the change of the difference information between frames, the brightness of the pedestrians, the edges of the stairs and the outlines of the trees will inevitably lose information. Compared with other methods, the method of the present application can well retain the brightness of the pedestrians and the scene details, and has a clearer visual effect, and the qualitative analysis result shows its advantages in retaining typical hot targets and rich visible details. Table 9 gives the average quantitative analysis results of the BEPM data set, and the method of the present application achieves the best value in the evaluation indexes of Q AB / F , Q0, Q e , MI and VIFF, ranks third in Q w , only less than A3 and A5, and only less than A5 on SSIM, these results all show that the method of the present application retains rich useful feature information and has better fusion performance than other methods, which is consistent with the above qualitative analysis.

[0114] Fusion results of the TNO data set

[0115] To demonstrate the robustness and effectiveness of the proposed method, it was further tested on the TNO dataset. Figure 12 shows five test fusion results obtained through different fusion methods. Regarding local details, the proposed method exhibits a stronger intensity distribution and more realistic and clearer textures on moving objects, such as pedestrians walking back and forth. Overall, the proposed method demonstrates superior performance in maintaining stronger similarity and correlation with the source image. Table 9 presents the quantitative experimental results. AB / F Q w The best values ​​were achieved in terms of Q0 and SSIM, with Q being the highest. e It ranks second only to A5 in terms of MI and VIFF. This invention is also quite good in terms of MI and VIFF.

[0116] Table 9. Quantitative evaluation index results for the BEPM and TNO datasets.

[0117] Furthermore, the computational efficiency of the method of this invention was also verified on the aforementioned dataset. All experiments were conducted on an Intel Core i7 PC equipped with a 3.3 GHz CPU and 4 GB of memory. The time complexity and running efficiency of different fusion methods are shown in Table 10. In summary, this invention achieves better fusion performance and higher computational efficiency.

[0118] Table 10 Performance Comparison of Various Fusion Methods

Claims

1. A method for fusing infrared and visible light video mimicry based on differential feature association synthesis and intuitionistic possibility sets, characterized in that: First, a differential feature set is constructed to quantitatively characterize the essential differences between infrared and visible light videos in terms of radiation characteristics, structural texture, and scene content. The K-nearest neighbor nonparametric estimation method is used to analyze the frequency distribution of feature amplitudes, and amplitude and frequency information are fused to construct a differential feature significance level to coordinate the competitive and complementary relationships among multiple attributes and objectively identify the significant differential features of each frame. Furthermore, a feature correlation matrix is ​​constructed using the Pearson correlation coefficient to explicitly characterize the intrinsic correlation between dissimilar features. Based on the element values ​​of the obtained correlation matrix, the corresponding T is selected. The operator performs distributed synthesis to calculate the differential feature association synthesis value corresponding to the feature association combination, thereby establishing differential feature significance association synthesis based on multi-rule combination; based on the fusion result of differential feature amplitude, an intuitive possible set partitioning criterion is established, and the high fusion effective interval is defined as the significant fusion information region; Finally, a fusion score index function is constructed, and the adaptive optimization of the fusion algorithm is achieved by combining the regional dominance rate and the average fusion effectiveness.

2. The infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possibility set according to claim 1, characterized in that: The process for identifying significant differences is as follows: The significance of differential features is defined as a dynamic function that characterizes the relative importance of different attributes in the entire image. , For the first In-frame difference features Significance of difference features , Let be the left and right boundaries of the feature set, respectively. For the difference characteristic amplitude The probability density distribution; A judgment criterion is constructed by introducing the golden ratio. In the formula, Represents video frames The significance of the difference feature in the video frame is used to determine whether the difference feature is significant. If the significance of the difference feature is greater than or equal to a threshold, then the difference feature is considered a video frame difference feature. Significant differences in characteristics.

3. The infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possibility set according to claim 2, characterized in that: The process of constructing the feature correlation matrix is ​​as follows: The feature correlation matrix is ​​constructed using the Pearson correlation coefficient. , In the formula, , Indicates differences in characteristics Significant differences were selected from the data. This represents a combination of features. Indicates significant differences Significant differences The Pearson correlation coefficient between them The total number of significantly different features; Pearson correlation coefficient The calculation is as follows: ,in, Indicates significant differences The fusion effectiveness score, Indicates significant differences The fusion effectiveness score; in, , In the formula, Indicates that variable A uses the fusion algorithm. k The result of the first Significant difference features of the frame-fused image in pixel block ξ amplitude, Indicates that variable A uses the fusion algorithm. k The result of the first Significant difference features of the frame-fused image in pixel block ξ amplitude, , They represent the first Significant differences in pixel block ξ between infrared and visible light source images. amplitude, , They represent the first Significant differences in pixel block ξ between infrared and visible light source images. The amplitude; Represents a similarity measure; , and , They represent significant differences. Significant differences The weighting coefficients.

4. The infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possibility set according to claim 3, characterized in that: The established high-integration effective range is , in, , .

5. The infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possibility set according to claim 4, characterized in that: The adaptive optimization process of the fusion algorithm is as follows: Fusion Algorithm Variable A k All feature associations involved are combined as follows ,definition Regional dominance rate , Variable A of the fusion algorithm k In related combinations The frequency of occurrence of significant fusion information regions For the fusion algorithm variable A k In related combinations The number of times the composite value of the difference features falls into the significant fusion information region. For all fusion algorithm variables in the associated combination The total number of times the composite value of the differential feature association falls into the significant fusion information region; Define average fusion efficiency , Variable A of the fusion algorithm k Average performance in areas with significantly integrated information. For the fusion algorithm variable A k The composite value of differential features when the j-th instance falls into the significant fusion information region; For each associated combination Calculate the variable A of the fusion algorithm k Fusion score After obtaining the comprehensive score of each fusion algorithm variable, the optimal fusion strategy is determined according to the maximization principle. ,in This represents the optimal fusion algorithm variable for the current frame.

6. The infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possible set according to claim 1, 2, 3, 4, or 5, characterized in that: The set of differential features includes six differential features: gray mean, standard deviation, average gradient, edge intensity, roughness, and contrast.

7. The infrared and visible light video mimicry fusion method based on differential feature association synthesis and intuitionistic possibility set according to claim 2, 3, 4, or 5, characterized in that: Difference characteristic amplitude , , They represent the first Differences between mid-frame infrared video and visible light video The amplitude on.

Images