A target recognition method based on visible light and infrared multimodal image fusion

Abstract

The present invention discloses a method for target recognition by fusing visible light and infrared multimodal images. The method first extracts features from visible light and infrared images, then constructs a hybrid graph neural network model, performs model information transmission and integration, and then updates the node states of the graph neural network model. Finally, multimodal data fusion features are output for target recognition. The method of the present invention combines the respective advantages of visible light and infrared, and obtains more comprehensive and complete features after fusing multimodal image data to improve the key performance of detection tasks such as weak target detection in complex backgrounds and false alarm suppression.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a visible light and infrared multimodal image fusion target recognition method based on a neural network. Background Art

[0002] As one of the most important branches of computer vision, object detection and recognition technology has been widely applied in recent years in fields such as scene understanding, medical-assisted diagnosis, autonomous driving systems, and military target monitoring. With the advent of high-definition imaging technology and high-resolution cameras, object detection and recognition based on visible light images has become a research hotspot. Maintaining robustness and stability in complex and ever-changing environments is a key requirement for object detection and recognition methods in a range of areas critical to socioeconomic and national security, including medical diagnosis, autonomous driving, and military reconnaissance.

[0003] Target detection and recognition based on single-modal image data obtained by visible light sensors faces great challenges in ensuring the stability and reliability of recognition performance when faced with complex scenes and adverse lighting and weather conditions, as the quality of images captured by the sensor depends largely on the environment and lighting conditions.

[0004] Multimodal information fusion has emerged to address the problem that the information obtained from single-modal image data cannot fully represent the object information, resulting in recognition performance indicators that cannot meet the needs of practical applications. Multimodal information fusion uses a specific method to effectively fuse the different feature information presented in images acquired by sensors in different spectral bands. It can maximize the complementary information of image data of different modalities and remove redundant information, thereby obtaining a more diverse and comprehensive representation of objects and improving recognition performance.

[0005] Sensors operating in the visible and infrared bands are widely used in numerous fields due to their low cost and rapid technological advancements. Visible light images offer high resolution and rich detail, but are susceptible to environmental influences. While infrared images have low contrast, they are resistant to interference and possess strong penetration capabilities, effectively addressing poor lighting conditions, visual obstructions such as occlusion, and adverse weather conditions such as rain, fog, and snow. Due to their complementary nature, effectively mining and fusing visible and infrared image data will facilitate scene understanding and enhance robust object recognition.

[0006] Current multimodal image fusion techniques can be categorized into low-level pixel-level image fusion methods and high-level feature-level image fusion methods. The significant differences in size and resolution between visible and infrared images make pixel-level registration difficult, making them unsuitable for pixel-level image data fusion. Feature-level image fusion of visible and infrared images involves fusing abstract image features extracted using traditional image feature extraction methods (such as principal component analysis, singular value decomposition, DSIFT, and empirical mode decomposition) and deep learning-based deep neural networks. However, features extracted from different modalities have different expressive properties, making existing feature fusion methods (such as vector concatenation and composite kernel methods) unable to address information interference during the fusion process, thus impacting the reliability and robustness of the recognition systems based on them. Summary of the Invention

[0007] In response to the information interference problem existing in the existing feature-level multimodal image data fusion method, this patent proposes a target detection and recognition method based on the fusion of visible light and infrared multimodal image data based on graph neural network.

[0008] The technical solution adopted by the present invention to solve the technical problem is: a method for target recognition by fusion of visible light and infrared multimodal images, comprising the following steps:

[0009] S1, extracting deep features of visible light and infrared images through graph neural networks based on deep learning methods: using sub-network N based on convolutional neural network model o From the visible light image X o Learning features of modality-specific image data Adopting the sub-network N based on the convolutional neural network model i From infrared image X i Learning features of modality-specific image data

[0010] S2, using the feature F extracted in step S1 o and F i Establish a hybrid directed graph neural model G=(V i ,V o ,E i ,E o ,E i→o ,E o→i ), where the graph nodes representing the infrared image features By formula The deep features F of the infrared image data extracted by the feature learning module i As the initialization feature of the graph node, the graph node representing the visible light image feature By formula The deep features F of the visible light image data extracted by the feature learning module o As the initialization feature of the graph nodes, the directed edges in the hybrid directed graph neural model can be used to represent the relationship between the graph nodes, where E i and E o They are used to represent the relationship between the nodes in the infrared image and visible light image modalities respectively:

[0011] S3, through the hybrid directed graph neural model established in step S2, the interaction and reasoning of intra-modal information and cross-modal information are carried out: According to the formula The information transmitted from node j to node k is obtained, and the nodes in the same mode complete the information aggregation through the edge vector connected to it; according to the formula The information transmitted from node j in modality s to node k in modality t is obtained, and the nodes across modalities complete information aggregation through the edge vectors connected to them. The edge vectors are mapped to the distribution weights of node features through Sigmoid (·) to complete the information aggregation between nodes.

[0012] S4, node state update of hybrid directed graph neural model: For intra-modal and cross-modal information, the node state is updated by integrating cross-modal information first and then integrating intra-modal information. The recurrent neural network (RNN) is used as the state update function. The cross-modal information is first integrated to obtain the updated intermediate state of node j belonging to modality t in the hybrid directed graph neural model. Then, based on the intermediate state, the information within the modality is integrated to obtain the new state of the node after the pth information aggregation iteration This completes the calculation of information interaction reasoning in subsequent iterative steps;

[0013] S5, output multimodal data fusion features for target recognition: through formula and Update the different modalities after P iterations of information transmission interaction and merge them to obtain the multimodal fusion feature H f ={H i ,H o The model output is compared with the actual label through the subsequent target detection and recognition module D(·). The loss calculated by the loss function is minimized to optimize the components of the recognition model: the feature extraction module, the multimodal feature fusion module based on the hybrid directed graph neural network, and the model parameters of the target detection and recognition module.

[0014] The method for target recognition by fusion of visible light and infrared multimodal images, step S2 of which adopts the relational function The correlation information between features within the same modality is extracted for the vectorized feature expression of directed edges, and the convolution operation Conv(·) of the edge vector is learned in a data-driven manner.

[0015] The method for target recognition by fusion of visible light and infrared multimodal images, step S2 is based on the formula and Modeling directed edges connecting nodes of different modalities and building a lightweight convolutional neural network Serves as a learnable domain transfer module ψ s→t The node expression of the source domain of (·) is converted into an adaptive expression adapted to the target domain: Here s, t∈{i, o} refer to the source domain and the target domain respectively, which are used to represent the correlation information of different modal features; through the relationship function Represents the correlation information of different modal features and vectorizes the directed edges that connect the relationships between cross-modal nodes.

[0016] The beneficial effects of the present invention are: the method of the present invention uses the outstanding advantages of the graph neural network model in relational reasoning and information interaction to establish high-level associations between complex features of infrared and visible light and complete information interaction, avoids the negative impact of information interference problems in the process of multimodal complex feature fusion, and improves the reliability and robustness of the recognition algorithm based on multimodal data fusion.

[0017] The method of the present invention combines the respective advantages of visible light and infrared, and obtains more comprehensive and complete features after fusing multimodal image data to improve the key performance of detection tasks such as weak target detection in complex backgrounds and false alarm suppression. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0019] The present invention discloses a method for target recognition by fusion of visible light and infrared multimodal images, which includes the following steps.

[0020] S1, visible light and infrared image feature extraction.

[0021] Extract visible light and infrared image features through a specific modality feature learning module: the feature learning module can use the feature extraction network in the commonly used classification model VGG or ResNet model, extract deep features based on the directed graph neural network of the deep learning method, and consists of two sub-networks N based on the convolutional neural network model. i and subnetwork N o They are used to extract the visible light image X o and infrared image X i Learn the features of the specific modality image data, namely:

[0022]

[0023]

[0024] S2, build a hybrid directed graph neural network model.

[0025] Use the features extracted in step S1 to build a hybrid directed graph neural model:

[0026] G=(V i ,V o ,E i ,E o ,E i→o ,E o→i ),

[0027] in It is a graph node that represents the features of infrared images. Here, the deep features F of the infrared image data extracted by the feature learning module are i As the initial feature expression of the graph node:

[0028]

[0029] The graph node that represents the features of the visible light image is the deep feature F of the visible light image data extracted by the feature learning module. o As the initial feature expression of the graph node:

[0030]

[0031] Directed edges in the hybrid directed graph neural model can be used to represent the relationships between graph nodes, where:

[0032]

[0033]

[0034] E i and E o They are used to represent the relationship between the nodes in the infrared image and visible light image modalities respectively.

[0035] This patent uses a relational function to extract the association information between features within the same modality for the vectorized feature expression of directed edges:

[0036]

[0037] The convolution operation Conv(·) of the edge vector is learned in a data-driven manner.

[0038] The hybrid directed graph neural model of this patent also needs to consider the connection between cross-modal nodes, and model it through directed edges connecting nodes of different modalities:

[0039]

[0040]

[0041] In order to fully explore the complementary information between different modal features and overcome the domain differences between cross-modal nodes, a lightweight convolutional neural network is constructed as a learnable domain transfer module ψ, which is different from the edge vectorization between nodes in the same modality. s→t (·) Convert the node expression of the source domain into an adaptive expression adapted to the target domain:

[0042]

[0043] Here s, t∈{i,o} refer to the source domain and target domain respectively.

[0044] The vectorized expression of directed edges used to represent the correlation information of different modal features and connect the relationships between cross-modal nodes is also extracted using a relational function:

[0045]

[0046] S3, model information transmission and integration.

[0047] The hybrid directed graph neural model established in step S2 is used to interact and reason about information within the same modality and information across modalities.

[0048] For intra-modal node information aggregation, nodes complete information aggregation through the edge vectors connected to them, and the information transmitted from node j to node k is:

[0049]

[0050] For cross-modal node information aggregation, nodes complete information aggregation through the edge vectors connected to them, and the information transmitted from node j in modality s to node k in modality t is:

[0051]

[0052] The edge vector is mapped to the distribution weight of the node feature through Sigmoid (·) to complete the information aggregation between nodes.

[0053] S4, node state update of hybrid directed graph neural model.

[0054] For both intra-modal and inter-modal information, a two-stage strategy is adopted here to update the node state: first integrating cross-modal information and then integrating intra-modal information: using the recurrent neural network RNN as the state update function, the updated intermediate state of node j belonging to modality t in the hybrid directed graph neural model is obtained by integrating cross-modal information:

[0055]

[0056] Then, based on the intermediate state, the information within the modality is integrated to obtain the new state of the node after the pth information aggregation iteration:

[0057]

[0058] Used for the calculation of information interaction reasoning in subsequent iterative steps.

[0059] S5, output multimodal data fusion features for target recognition.

[0060] After P iterations of information transmission interaction, the updated features of different modalities are expressed as follows: Merge to obtain multimodal fusion features H f ={H i ,H o}.

[0061] The fused features are used in the subsequent target detection and recognition module D(·). The target detection and recognition module D(·) can use the detector head module in the classic detection model Yolo, compare the model output results with the actual labels, and optimize the various components of the recognition model by minimizing the loss calculated by the loss function: the feature extraction module, the multimodal feature fusion module based on the hybrid graph neural network, and the model parameters of the target detection and recognition module.

[0062] The specific details of steps S1 to S5 are given by Figure 1 shown.

[0063] The above description is only for the purpose of illustrating the technical solution and is not intended to limit the same. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment. Those skilled in the art should understand that any modification or equivalent substitution that does not depart from the spirit and scope of the present invention is included in the scope of the claims of the present invention.

Claims

1. A method for target recognition by fusion of visible light and infrared multimodal images, characterized by: Includes the following steps S1, using sub-network N based on convolutional neural network model o From the visible light image X o Learning features of modality-specific image data Adopting the sub-network N based on the convolutional neural network model i From infrared image X i Learning features of modality-specific image data S2, using feature F o and F i Establish a hybrid directed graph neural model G=(V i ,V o ,E i ,E o ,E i→o ,E o→i ), where the graph nodes representing the infrared image features By formula The deep features F of the infrared image data extracted by the feature learning module i As the initialization feature of the graph node, the graph node that represents the visible light image feature By formula The deep features F of the visible light image data extracted by the feature learning module o As the initialization feature of the graph nodes, the directed edges in the hybrid directed graph neural model are used to represent the relationship between the graph nodes, where E i and E o They are used to represent the relationship between the nodes in the infrared image and visible light image modalities respectively: S3, interaction and reasoning of intra-modal and cross-modal information through a hybrid directed graph neural model: According to the formula The information transmitted from node j to node k is obtained, and the nodes in the same mode complete the information aggregation through the edge vector connected to it; according to the formula The information transmitted from node j in modality s to node k in modality t is obtained, and the nodes across modalities complete information aggregation through the edge vectors connected to them. The edge vectors are mapped to the distribution weights of node features through Sigmoid (·) to complete the information aggregation between nodes. S4, using the recurrent neural network RNN as the state update function, first integrates the cross-modal information to obtain the updated intermediate state of node j belonging to modality t in the hybrid directed graph neural model Then, based on the intermediate state, the information within the modality is integrated to obtain the new state of the node after the pth information aggregation iteration. This completes the calculation of information interaction reasoning in subsequent iterative steps; S5, output multimodal data fusion features for target recognition: through formula and After P times of information transmission interaction iteration, different modalities are updated and merged to obtain the multimodal fusion feature Hf = {H i ,H o The model output is compared with the actual label through the subsequent target detection and recognition module D(·), and the model parameters of the recognition feature extraction module, the multimodal feature fusion module based on the hybrid directed graph neural network, and the target detection and recognition module are optimized by minimizing the loss calculated by the loss function.

2. The method for target recognition by fusion of visible light and infrared multimodal images according to claim 1, characterized in that: The step S2 adopts the relation function The correlation information between features within the same modality is extracted for the vectorized feature expression of directed edges, and the convolution operation Conv(·) of the edge vector is learned in a data-driven manner.

3. The method for target recognition by fusion of visible light and infrared multimodal images according to claim 1, characterized in that: Step S2 is as follows: and Modeling directed edges connecting nodes of different modalities to build a lightweight convolutional neural network For learnable domain transfer module ψ s→t (·) is converted from the source domain to the target domain, where s, t∈{i,o} represent the source domain and the target domain respectively; through the relation function Represents the correlation information of different modal features and vectorizes the directed edges that connect the relationships between cross-modal nodes.

Images