Infrared and visible light video fusion algorithm based on multi-agent collaboration
Through the multi-agent collaborative fusion algorithm, a fusion agent is built to dynamically adjust the fusion parameters and structure, which solves the problem that the fusion strategy in the existing technology cannot adapt to the different changes in videos, realizes the efficient fusion of infrared and visible videos, and improves the fusion quality and system stability.
Patent Information
- Application Number
- CN202411055431.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-15
AI Technical Summary
The existing infrared and visible video fusion methods cannot dynamically adjust the fusion strategy according to the differences between videos, resulting in poor fusion effect or failure, making it difficult to adapt to dynamically changing detection scenarios and imaging environments.
Using infrared and visible video fusion algorithm based on multi-agent collaboration, we construct fusion agents of different feature types. Through collaboration between agents, we select the best fusion parameters, rules and algorithms, and dynamically adjust the fusion structure to achieve the difference in video frames. Adaptive fusion of features.
The fusion quality of infrared and visible videos is significantly improved, the robustness, reliability and flexibility of the system are enhanced, the fusion strategy can be dynamically adjusted according to the changes in video frames, and the comprehensive detection and accurate interpretation of video features is improved.
Smart Images

Figure CN118967477A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an infrared and visible light video fusion method, in particular to an infrared and visible light video fusion algorithm based on multi-agent collaboration. Background Art
[0002] Infrared and visible light video sensing technologies detect the essential characteristics and differences of scene targets by utilizing different radiation physical properties and imaging methods. The two types of videos are significantly complementary. The fusion of the two videos maximizes the imaging advantages of each channel and greatly improves the ability to comprehensively detect and accurately interpret video features. This technology is crucial for identifying targets of interest in applications such as situational awareness in degraded visual environments, security monitoring and early warning of entrance areas, precise positioning of high-speed moving targets, and intelligent obstacle avoidance.
[0003] At present, the methods of infrared and visible light video fusion are mainly divided into two categories: the first category is static single-frame fusion methods, which usually use multi-scale transformation, neural network, sparse representation and hybrid technology, but they are difficult to meet the real-time requirements; the second category is based on multi-frame spatiotemporal information fusion methods, which are realized through three-dimensional non-separable transformations such as spatiotemporal energy matching, spatiotemporal structure tensor, and high-order singular value decomposition. It can effectively consider the spatial geometry and time dimension information between consecutive frames, thereby maintaining the temporal stability and consistency of the fused video. However, the existing fusion strategies mainly rely on the prior knowledge of the video. In practical applications, due to the dynamics of the detection scene and the variability of the imaging environment, especially the more complex attributes such as the type, amplitude, frequency and other attributes of the difference feature in the frame of the video sequence and their changes, the pre-determined fusion model is difficult to adapt to the needs of dynamic changes. Therefore, the traditional method has become a bottleneck for improving the fusion effect of the two videos.
[0004] The study found that the agent model can well meet the architectural requirements of hierarchical fusion. The deliberative agent is the most intelligent agent. It can make rigorous reasoning and thinking based on its own understanding of the environment and its own will, and make detailed action plans for collaboration and tasks, which is consistent with the requirements of the selection process of the optimal fusion strategy in video fusion; the self-learning agent can update its knowledge base according to environmental changes, and its performance continues to improve over time, which is consistent with the characteristics of storing the fusion strategy knowledge base during the fusion process; the reactive agent can respond quickly to external information, and can use its rapid responsiveness to directly call the fusion strategy for the existing knowledge base. Under the multi-scale fusion framework, the multi-agent collaborative fusion algorithm realizes the complex functions of dynamic selection of fusion parameters, rules and algorithms through the collaboration of simple agents, while reducing the complexity of the system and improving the robustness, reliability and flexibility of the system.
[0005] In summary, the intelligent agent can autonomously perceive changes in the external environment and respond accurately. By cooperating with other intelligent agents, it can develop behavioral strategies suitable for completing complex and comprehensive tasks, providing a theoretical tool for the efficient selection of fusion strategies. Summary of the invention
[0006] Aiming at the problem that the existing fusion model cannot dynamically adjust the fusion strategy according to the difference between videos and ensure the full play of the fusion performance, the present invention proposes an infrared and visible light video fusion algorithm based on multi-agent collaboration, constructs fusion agents of different feature types, selects the best fusion parameters, rules and algorithms within the agent, and obtains the best fusion structure through interactive collaboration between agents, solving the problem that the traditional model cannot dynamically change the fusion algorithm, fusion rules, fusion parameters and structure. In this way, the complementary advantages of the two types of imaging are combined to provide a new idea for the fusion of infrared and visible light videos.
[0007] The present invention is implemented by adopting the following technical solution: an infrared and visible light video fusion algorithm based on multi-agent collaboration, comprising the following steps:
[0008] Fusion agents are constructed for different features of infrared and visible light images. Each fusion agent includes fusion algorithm, fusion rules and fusion parameters.
[0009] Determine the main difference features of each frame in infrared and visible light video sequences;
[0010] The fusion agent corresponding to the main difference feature of each frame image is optimized. The fusion agent first fixes the fusion rules and fusion parameters, compares the fusion effects of different fusion algorithms, and infers and decides the optimal fusion algorithm corresponding to the video frame to be fused; the fusion agent selects the optimal fusion algorithm for each frame, and combines the fusion effectiveness to determine the optimal fusion rule corresponding to the main difference feature of each frame. First, the low-frequency fusion rule and fusion parameters are fixed, and all high-frequency fusion rule combinations corresponding to the main difference features are fused. Then, the fusion parameters and the previously determined high-frequency fusion rules are fixed, and the low-frequency fusion rules are fused. Finally, with the fusion parameters as variables, the fusion agent compares the fusion effects of each fusion parameter in turn according to the fusion effectiveness.
[0011] After the fusion agent corresponding to the main difference features of each frame of the infrared and visible light video sequences is optimized, the fusion of the infrared and visible light video sequences can be performed.
[0012] The above-mentioned infrared and visible light video fusion algorithm based on multi-agent collaboration selects six types of features, namely grayscale mean, edge intensity, standard deviation, average gradient, roughness and contrast, to measure the difference between the two types of videos.
[0013] In the above infrared and visible light video fusion algorithm based on multi-agent collaboration, the process of determining the main difference features of each frame image is as follows:
[0014] Where D i,m The difference feature representing the mth feature of the i-th frame of the infrared and visible light video sequences; The mth feature of the ith frame of the infrared and visible light video sequences respectively;
[0015] Normalize the difference features of each frame.
[0016] In the formula, Represents the normalized value of the mth difference feature; the feature type corresponding to the maximum value of the normalized data is the main difference feature of the frame.
[0017] The above-mentioned infrared and visible light video fusion algorithm based on multi-agent collaboration selects frames with significant changes as significant frames in each frame of the infrared image video sequence or the visible light image video sequence, only determines the main difference features of the significant frame images in the infrared and visible light video sequences, and optimizes the fusion agent corresponding to the main difference features of the significant frame images.
[0018] The above infrared and visible light video fusion algorithm based on multi-agent collaboration, the fusion effectiveness is defined as:
[0019] in:
[0020]
[0021] Where: represents the mth feature of the i-th frame after the infrared and visible light video sequences are fused, represents the fusion effectiveness of the mth feature of the i-th frame, and are the weights of infrared and visible light video frames respectively.
[0022] The above-mentioned infrared and visible light video fusion algorithm based on multi-agent collaboration, for the case where there is more than one main difference feature in a frame image, forms different fusion structures by fusing the agents in series, parallel, and embedded combinations, and then calculates the fusion effectiveness under different fusion structures to select the optimal fusion structure.
[0023] The present invention proposes a method for fusion of infrared and visible light video based on multi-agent collaboration, and constructs a distributed multi-agent video fusion system. A single fusion agent in the system selects the optimal fusion strategy for the corresponding main difference feature based on the fusion effectiveness function. For the case where there is more than one main difference feature, the system selects the optimal fusion structure through collaborative interaction between agents. This method of achieving adaptive fusion based on the combination of fusion algorithms, rules, parameters and structural changes significantly improves the fusion quality of infrared and visible light videos, and solves the problem that the existing fusion model cannot be dynamically adjusted according to the difference features of the video frames, resulting in poor fusion effect or even fusion failure. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is the structural block diagram of the grayscale mean fusion agent.
[0025] Figure 2 Schematic diagram of the multi-agent collaborative fusion model.
[0026] Figure 3 Schematic diagram of the salient frames of infrared and visible light videos in the BEPMS dataset, from left to right: frame 1, frame 12, frame 50, frame 105, frame 185, and frame 200.
[0027] Figure 4 Comparison chart of fusion effectiveness of different fusion algorithms for each salient frame.
[0028] Figure 5 The figure is a comparison of the fusion performance of different fusion structures.
[0029] Figure 6 The fusion results of some significant frames in the BEPMS dataset are as follows: (a) frame 1; (b) frame 65; (c) frame 185; each frame from left to right are: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, GFF, WPT, NestFuse (avg), NestFuse (max) and the result of the method of the present invention.
[0030] Figure 7 The fusion results of some significant frames in the OTCBVS dataset are as follows: (a) the 1st frame; (b) the 60th frame; (c) the 75th frame; each frame from left to right is: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, GFF, WPT, NestFuse (avg), NestFuse (max) and the result of the method of the present invention.
[0031] Figure 8These are the fusion result diagrams of some significant frames in the INO dataset, (a) frame 360; (b) frame 405; (c) frame 502; each frame from left to right are: infrared image, visible light image, CVT, DTCWT, LP, NSCT, NSST, GFF, WPT, NestFuse (avg), NestFuse (max) and the result of the method of the present invention. DETAILED DESCRIPTION
[0032] 1. Construction of Fusion Agent
[0033] In the video fusion model composed of intelligent agents, the intelligent agents mainly realize two functions: one is to make local plans and execute them for their own goals, and the other is to cooperate with other intelligent agents to achieve the system goals. From the perspective of the functions of the intelligent agents, it is necessary to design an internal structure that is compatible with the functions of the intelligent agents.
[0034] According to the fusion requirements, encapsulating or mapping logical functions into agents determines the granularity of the agents. Too large a granularity will cause the internal structure of the agent to be too bloated and inefficient; too small a granularity will cause too many agents and a complex system. Therefore, the size of the agent granularity design affects the execution efficiency of the entire multi-agent system. The present invention designs 6 agents according to feature categories, combining the characteristics of reactive agents, deliberate agents, and self-learning agents, and naming them according to the difference feature types. Their function is to select the optimal algorithm, rules, and parameters in the video fusion process.
[0035] Taking the gray mean feature fusion agent as an example, the internal design of the fusion agent is introduced in detail, as shown in the figure:
[0036] 1.1 Representation and selection of main difference features
[0037] There are significant differences in the imaging mechanism and image characteristics of infrared and visible light sensors. Infrared videos generally contain more thermal target information and the outlines presented are more obvious. Visible light videos contain richer detail information. The brightness, edge and texture features of the two types of image features are significantly different. Therefore, six types of features, including gray mean (GM), edge intensity (EI), standard deviation (SD), average gradient (AG), coarseness (CA) and contrast (CN), can be selected to measure the difference between the two types of videos.
[0038] The difference feature represents the absolute difference between the features in the corresponding frames of the two types of videos, as shown in formula (1), D i,m The difference feature representing the mth feature of the i-th frame of the infrared and visible light video sequences; They represent the mth feature of the i-th frame of the infrared and visible light video sequences respectively.
[0039]
[0040] Since the sizes of the six types of difference features often have different dimensions, which affects the results of data analysis, in order to enhance the comparability between features and facilitate the selection of the main difference features of each frame, the difference features of each frame are normalized (see formula (2)), and the value range is [0, 1]. The feature type corresponding to the maximum value of the normalized data is the main difference feature of the frame.
[0041]
[0042] In the formula, Represents the normalized value of the mth difference feature.
[0043] 1.2 Fusion Effectiveness
[0044] To determine the optimal combination of fusion algorithms corresponding to the difference features, in addition to visual pros and cons, objective evaluation is also required. In the case of undistorted images, the higher the difference feature value, the better the image quality. Since direct comparison of the difference feature value is not intuitive, in order to more conveniently observe the fusion effect of the difference features of the dual-modal infrared image, the fusion effectiveness is used to describe the fusion of the difference features under different algorithm combinations, so as to measure the fusion effect of the difference features of the infrared and visible light images, and select the strategy with the best fusion effect of the main difference features. As shown in formula (3), the larger the value, the higher the fusion effectiveness.
[0045]
[0046] in:
[0047]
[0048] Where: represents the mth feature of the i-th frame after the infrared and visible light video sequences are fused, represents the fusion effectiveness of the mth feature of the i-th frame, and are the weights of infrared and visible light video frames respectively.
[0049] 1.3 Fusion Agent Workflow
[0050] The main difference feature of the video frames to be fused is determined as the gray mean, and the gray mean agent selects the fusion algorithm, rules and parameters. Deliberative agents are suitable for handling these situations that require analysis and decision-making. The deliberative characteristics of the agent are explained through the belief and desire of the image fusion process. In this process, belief is defined as the agent's internal representation of the state of the environment, including the cognition of the characteristics of infrared and visible light images and the planning of fusion strategies. The desire describes the goal that the agent hopes to achieve through the fusion process, such as selecting a fusion algorithm that maximizes the fusion effectiveness value of the fused image. According to the interpretation results, the fusion algorithm set, fusion rule set and fusion parameter set are called, and the fusion strategy with the largest fusion effectiveness value is selected in combination with the fusion effectiveness function. Then, the self-learning characteristics of the agent are used to store it in the fusion strategy knowledge base.
[0051] 2 Construction of multi-agent collaborative fusion model
[0052] In the multi-scale fusion framework, the fusion model dynamically responds to the complex and changing difference features within or between video frames to meet the complex fusion requirements of different scenarios, similar to the independent operation of agents within a multi-agent system or the interactive collaboration with other agents. The fusion agent adjusts the fusion algorithm, rules and parameters through internal decision-making, and different agents assist in adjusting the fusion structure through interaction. The fusion model dynamically responds to the complex and changing difference features within or between video frames to meet the complex fusion requirements of different scenarios.
[0053] The multi-agent collaborative infrared and visible light video fusion proposed in the present invention is to establish the connection between multi-agent characteristics and video fusion, construct a fusion model with flexible and variable structure according to the fusion requirements, and realize a fusion method that can dynamically adjust and effectively select fusion algorithms, fusion rules, fusion parameters and fusion structures according to the changes in difference characteristics. The main process is:
[0054] 2.1 Multi-agent collaborative fusion
[0055] First, in each frame of the video sequence, the region of interest is divided, and the frames with significant changes are selected as significant frames. Then, the difference features of these significant frames are calculated in detail to identify the main difference features of each frame; secondly, a fusion effectiveness function is constructed, through which the fusion effectiveness values of different fusion strategies are calculated to determine the optimal fusion strategy; finally, a fusion agent based on difference features is built, and on the basis of the interactive collaboration of multiple fusion agents, a multi-layer nested and dynamically variable fusion system of fusion parameters, rules and algorithms is constructed to ensure the optimal selection of the fusion strategy and realize the dynamic adjustment of the algorithm in the process of infrared and visible light video fusion.
[0056] The multi-agent collaborative fusion system adopts a distributed organizational structure and builds fusion agents based on the responsive, deliberative and self-learning characteristics of agents. In the system, all decisions are made by negotiation between agents, which gives the system a high degree of robustness and scalability. For the main difference features of each significant frame, the system will select the corresponding fusion agent. Each agent is both independent and interrelated, and its internal fusion algorithm, rules and parameters are dynamically adjusted according to the actual fusion needs. Through the interactive collaboration between agents, the system can screen out the optimal fusion structure.
[0057] 2.2 Characterization of multi-scale fusion methods
[0058] The fusion algorithm, rules, parameters and structure sets are introduced below.
[0059] Fusion algorithm collection: The present invention selects 7 algorithms in the multi-scale fusion framework, namely, curvelet transform (CVT), dual-tree complex wavelet transform (DTCWT), Laplace pyramid transform (LP), non-subsampled contourlet transform (NSCT), non-subsampled shearlet transform (NSST), guided filtering (GFF) and wavelet packet transform (WPT), which are represented by A1-A7 respectively.
[0060] Fusion rule set: Fusion rule is the core of video fusion. The fusion rule based on the multi-scale fusion framework of the present invention is mainly divided into high-frequency fusion rule and low-frequency fusion rule. As shown in Table 1, the low-frequency and high-frequency fusion rules can be combined arbitrarily.
[0061] Table 1 Fusion rule category set.
[0062]
[0063] Fusion parameter set: The fusion parameters of the present invention mainly refer to the number of multi-scale decomposition layers of the algorithm and the type of internal filter. For example, LP, DTCWT, etc. have layered parameters n, filters 'near_sym_a',
[0064] 'antonini', 'legall'; the direction parameter (dcomp) and scale parameter (dsize) of shear waves in the NSST algorithm, 'pyrexc', 'pyr', 'maxflat', '7-9', etc. of pyramid transformation.
[0065] Fusion structure class: Fusion structure variants include three types: tandem structure (TS), parallel structure (PS) and embedded structure (ES). (a) Tandem: The output of the previous fusion algorithm is used as the input of the next fusion algorithm, and the reasonable design of the input of each fusion algorithm after the second level must be guaranteed. (b) Parallel: Different fusion algorithms have the same input, and the images are fused at the same time. Finally, the different fusion results are synthesized to obtain the fusion result of the parallel structure. (c) Nested: Different fusion algorithms are nested layer by layer to obtain the final fusion result. The internal output can be used as the input of other fusion algorithms, and other fusion algorithms can be embedded in them. When the internal fusion is completed, the external fusion algorithm combines the different internal fusion results to obtain the fusion result of the embedded structure.
[0066] 3 Experimental simulation
[0067] 3.1 Source Video Dataset
[0068] The present invention conducts experiments on three public datasets to demonstrate the advantages and robustness of the proposed method. The first dataset is BEPMS, which involves a video of a soldier wearing camouflage uniforms walking in a dense forest, where only the first 200 frames are selected as the research object, and the image size is 576×480. The second dataset is the OTCBVS dataset, which contains 17089 image pairs of different scenes, where the selected image pairs are from the OSU Color-Thermal change video. The third dataset is the INO dataset, and the selected video frames are from the ParkingEvening video.
[0069] 3.2 Experimental Results
[0070] The method is illustrated by using infrared and visible light video sequences selected from the BEPMS dataset. First, the region of interest is roughly divided in each frame of the video sequence, and the salient frames 1, 65, 120, 140, 185, and 200 are selected through human eye screening and data analysis, a total of 6 frames for display; Figure 3 As shown in the red frame; Formula (1) is then used to calculate each difference feature of the region of interest in each group of video frames. In order to enhance the comparability between the features, the numbers are normalized according to Formula (2) (as shown in Table 2); thus, the main difference feature corresponding to each frame is obtained.
[0071] Table 2 Normalized results of significant frame difference features.
[0072]
[0073] Figure 4 The fusion effectiveness distribution of different fusion algorithms corresponding to the main difference features of each significant frame is shown in Figure 2. For example, the main difference feature corresponding to the first frame is EI, and the corresponding optimal algorithm is A2 (DTCWT). The main difference features corresponding to the fusion effectiveness comparison diagram of the 185th frame are AG, SD and EI from left to right, and the corresponding optimal algorithms are NSCT, WPT and DTCWT. Similarly, the optimal fusion algorithms corresponding to the main difference features of the remaining significant frames can be obtained. On the basis of selecting the optimal fusion algorithm for each significant frame, different fusion rules and fusion parameters are used as variables, and the fusion effects of fusion rules and fusion parameters are compared in turn according to the fusion effectiveness function. The analysis process is the same as above.
[0074] Then, according to the fusion effectiveness formula, the fusion effects of different fusion strategies corresponding to the main difference features in each frame of video under the fusion algorithm, fusion parameters, fusion rules and fusion structure are calculated respectively, and the fusion strategy is optimized according to the main difference features of each frame to obtain the final fusion result. Thus, the optimal fusion algorithm, fusion rule and fusion parameter corresponding to each frame can be obtained, and the results are shown in Table 3.
[0075] Table 3 The optimal fusion algorithmcombination selected by the differential feature agent.
[0076]
[0077] Combinations such as series, parallel, and embedded will form different fusion structures. When there is more than one main difference feature in a frame of a video sequence, the optimal fusion strategy selected by the corresponding fusion agent is different. At this time, it is necessary to combine different fusion strategies through the interactive collaboration of several fusion agents. For example, in the 185th frame of the above video sequence, there are three types of main difference features SD, AG, and EI. Consider the fusion effect of the corresponding fusion structure of the collaborative combination of these three fusion agents.
[0078] The experiments of three fusion structures are as follows: Series: First, use the standard deviation fusion agent to fuse the image to be fused, then use the edge intensity fusion agent to perceive the fusion result and the infrared video frame to continue to obtain the fusion result, and finally use the average gradient fusion agent to perceive the fusion result of the second step and the visible light video frame to obtain the final fusion result. Parallel: The fusion results of the three fusion agents are averaged and weighted to obtain the final fusion result. Embedded: The fusion algorithm of the standard deviation fusion agent is used to decompose the image to be fused to obtain high-frequency coefficients and low-frequency coefficients. The low-frequency coefficients are decomposed by the EI fusion agent, and the high-frequency coefficients are decomposed by the AG fusion agent. Finally, the final fusion result is obtained by layer-by-layer reconstruction.
[0079] For the 185th frame, the fusion effectiveness of different difference features under three types of fusion structures, namely series, parallel and embedded, is calculated, and the optimal fusion structure variable is selected as the series type, see Figure 5 .
[0080] 4 Experimental results analysis
[0081] In order to verify the rationality and effectiveness of the multi-agent collaborative fusion method proposed in the present invention, the method of the present invention is compared with the selected multi-scale fusion method and the NestFuse fusion model based on deep learning. The fusion rules all adopt the large absolute value of high frequency and the weighted average of low frequency.
[0082] Based on the optimal fusion strategy selected above, the fusion results of each frame are discussed. AB / F (measures edge information obtained from the source image), Q w (visual saliency detection based on window weights), Q0 (measurement of structural distortion of fusion results), Q e (Based on Q w The fusion results of each algorithm are evaluated by seven objective evaluation indicators: (i) saliency (Saliency), (ii) mutual information (MI), (iii) visual information fidelity (VIFF), and (iv) spatial frequency (SF). The higher the indicator value, the better the fusion performance. In the experiment, the optimal and suboptimal values are highlighted in red and black bold.
[0083] Table 4 Results of each evaluation index of the BEPMS dataset.
[0084]
[0085] Table 5 Results of each evaluation index of the OTCBVS dataset.
[0086]
[0087]
[0088] Table 6 Results of each evaluation index of the INO dataset.
[0089]
[0090] Qualitative and quantitative experiments were conducted on the BEPMS dataset, OTCBVS dataset, and INO dataset. From the results of several frame fusions shown, it can be seen that a typical single fusion algorithm cannot maintain the best fusion effect on all video sequences. As the difference in information within the frame or the content of each frame changes, the fusion performance of the algorithm will also change, and the brightness of pedestrians, the edges of stairs, and the outlines of trees will inevitably lose information. This method dynamically adjusts the fusion algorithm according to the changes in the difference characteristics of the region of interest, optimizes the selection of fusion strategies, and thus retains the brightness of pedestrians and scene details to the greatest extent. Compared with other methods, it has a clearer visual effect. The qualitative analysis results prove its advantages in retaining typical thermal targets and rich visible details.
[0091] Table 4 shows the quantitative analysis results of the BEPMS dataset. AB / F , Q e , Q0 and SF, and reaches the maximum value in Q w It ranks second only to CVT and NestFuse(avg) in MI and ranks third in VIFF. These results all show that this method retains rich effective feature information in the fused image and has better fusion performance than other methods, which is consistent with the above qualitative analysis. Table 5 gives the quantitative experimental results of the OTCBVS dataset. AB / F , Q w , Q e , VIFF and SF, and ranks second only to NestFuse(avg) in MI. Table 6 shows the quantitative experimental results of the INO dataset. AB / F , Q w, Q0, MI and SF. Based on the fusion results of the three data sets, although the maximum values were not obtained under all the evaluation indicators, the optimal values of the evaluation indicators of the fusion results obtained by this method accounted for 66.7% of the total, and the suboptimal values accounted for 80.9% of the total. The fusion results of the BEPMS, OTCBVS and INO_ParkingEvening data sets show that the method proposed in the present invention retains the typical infrared targets and visible light structure details in the video as a whole, realizes the effective fusion of difference information and the adaptability of image fusion, and the fusion performance is significantly better than other single fusion methods.
Claims
1. Infrared and visible light video fusion algorithm based on multi-agent collaboration, characterized by: The following steps are involved: Fusion agents are constructed for different features of infrared and visible light images. Each fusion agent includes fusion algorithm, fusion rules and fusion parameters. Determine the main difference features of each frame in infrared and visible light video sequences; The fusion agent corresponding to the main difference feature of each frame image is optimized. The fusion agent first fixes the fusion rules and fusion parameters, compares the fusion effects of different fusion algorithms, and infers and decides the optimal fusion algorithm corresponding to the video frame to be fused; the fusion agent selects the optimal fusion algorithm for each frame, and combines the fusion effectiveness to determine the optimal fusion rule corresponding to the main difference feature of each frame. First, the low-frequency fusion rule and fusion parameters are fixed, and all high-frequency fusion rule combinations corresponding to the main difference features are fused. Then, the fusion parameters and the previously determined high-frequency fusion rules are fixed, and the low-frequency fusion rules are fused. Finally, with the fusion parameters as variables, the fusion agent compares the fusion effects of each fusion parameter in turn according to the fusion effectiveness. After the fusion agent corresponding to the main difference features of each frame of the infrared and visible light video sequences is optimized, the fusion of the infrared and visible light video sequences can be performed.
2. The infrared and visible light video fusion algorithm based on multi-agent collaboration according to claim 1 is characterized in that: Six features including grayscale mean, edge strength, standard deviation, average gradient, roughness and contrast are selected to measure the difference between the two types of videos.
3. The infrared and visible light video fusion algorithm based on multi-agent collaboration according to claim 1 or 2 is characterized in that: The process of determining the main difference features of each frame image is: Where D i,m The difference feature representing the mth feature of the i-th frame of the infrared and visible light video sequences; The mth feature of the ith frame of the infrared and visible light video sequences respectively; Normalize the difference features of each frame. In the formula, Represents the normalized value of the mth difference feature; the feature type corresponding to the maximum value of the normalized data is the main difference feature of the frame.
4. The infrared and visible light video fusion algorithm based on multi-agent collaboration according to claim 1 or 2 is characterized in that: In each frame of the infrared image video sequence or the visible light image video sequence, the frame with significant changes is selected as the significant frame, only the main difference features of the significant frame images in the infrared and visible light video sequences are determined, and the fusion intelligent body corresponding to the main difference features of the significant frame images is optimized.
5. The infrared and visible light video fusion algorithm based on multi-agent collaboration according to claim 1 or 2 is characterized in that: Fusion effectiveness is defined as: in: Where: represents the mth feature of the i-th frame after the infrared and visible light video sequences are fused, represents the fusion effectiveness of the mth feature of the i-th frame, and are the weights of infrared and visible light video frames respectively.
6. The infrared and visible light video fusion algorithm based on multi-agent collaboration according to claim 1 or 2 is characterized in that: In case that there is more than one main difference feature in a frame image, different fusion structures are formed by fusing intelligent agents in series, parallel and embedded combinations. The fusion effectiveness under different fusion structures is then calculated to select the optimal fusion structure.
Citation Information
Patent Citations
Infrared video and visible light video adaptive fusion method
CN107784642A
Mimicry hierarchical fusion method for infrared and visible light videos
CN114998574A
Policy decision-making method in multi-agent collaboration and confrontation
CN116167415A
Multi-agent game decision-making system and method based on cloud native
CN117076993A
