Target segmentation identification method based on visible light and infrared modal fusion

Abstract

The invention provides a target segmentation recognition method based on visible light and infrared modal fusion, which comprises the following steps of: 1, acquiring an infrared image and recording the infrared image as ii, and recording a visible light image as iv; step 2, performing registration preprocessing on the infrared image as ii and the visible light image as iv to obtain an infrared registration image as iir and a visible light registration image as ivis; step 3, pre-training the fusion network; 4, recording the infrared registration image as iir and the visible light registration image as ivis, inputting the images into a fusion network for fusion, and outputting a fusion image; 5, carrying out semantic segmentation on the fused image, and carrying out target identification and positioning; according to the method, the calculation pressure of a multi-source perception fusion module is reduced, and the dual-light image fusion efficiency based on the semantic fusion network is improved. Moreover, loss function limitation of manual subjective design can be avoided, the high-quality visual effect is guaranteed, the performance of advanced visual tasks is remarkably improved, and meanwhile the application value and effect of the image fusion technology are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of infrared image processing, and in particular relates to a target segmentation and recognition method based on visible light and infrared modality fusion. Background Art

[0002] Visible and infrared image fusion (IVIF) technology is an important branch in the field of computer vision. It aims to combine the advantages of multi-source images, break through the limitation of single sensor information loss, and play a key role in advanced visual tasks such as target detection and semantic segmentation. Early image fusion methods relied on manually designed fusion rules to approximate aggregated modal features. However, manually designed fusion rules cannot adapt to various complex and dynamically changing visual scenes, and are prone to artifacts and blurring. More importantly, it is difficult to retain modal features suitable for subsequent advanced visual tasks.

[0003] In order to solve the limitations of manual rules, researchers proposed an end-to-end learning method, and image fusion technology began to apply complex algorithm models to adaptively learn fusion strategies. In the past, most fusion methods only focused on improving visual effects, and often did not work well in more advanced visual tasks. Summary of the invention

[0004] The purpose of the present invention is to combine the advantages of existing image enhancement algorithms and improve them, and to propose a target segmentation and recognition method based on the fusion of visible light and infrared modalities.

[0005] To this end, the present invention provides a target segmentation and recognition method based on visible light and infrared modality fusion, comprising the following steps:

[0006] Step 1: Get the infrared image and record it as i i , the visible light image is recorded as i v ; Infrared images and visible light images can be collected by corresponding sensors;

[0007] Step 2: The infrared image is recorded as i i , the visible light image is recorded as i v Perform registration preprocessing to obtain the infrared registration image, denoted as i ir , the visible light registration image is recorded as i vis ;

[0008] Step 3: Pre-train the fusion network;

[0009] Step 4: record the infrared registration image as i ir , the visible light registration image is recorded as i vis Input fusion network for fusion, output fusion image i f ;

[0010] Step 5: fusion image i f Perform semantic segmentation and identify and locate the target Obj.

[0011] Furthermore, in step 2, the infrared image is recorded as i i , the visible light image is recorded as i v Perform registration preprocessing to obtain the infrared registration image, denoted as i ir , the visible light registration image is denoted as ix is The specific process is:

[0012] Step 201, calculating the image relative parameter transformation matrix;

[0013] For infrared images, denoted as i i , the visible light image is denoted as i v Find N pairs of feature matching points from two multi-source images, and the infrared image is denoted as i i , the visible light image is recorded as i v The relative parameter transformation matrix P between I→V Calculated using the least squares formula:

[0014] P I→V =V*I T *(I*I T ) -1 (1)

[0015]

[0016] Where I represents the coordinates of the matching points of the infrared image as a 3×N matrix, V represents the coordinates of the matching points of the visible light image as a 3×N matrix, and I T is the transposed matrix of I, the relative parameter transformation matrix P I→V is a 3×3 matrix;

[0017] Step 202, transforming the visible light into the infrared image coordinate system;

[0018] According to the infrared image i Midpoint I 0 The coordinates of the reference image i are calculated i The coordinates of the corresponding point V 0 , and its calculation formula is:

[0019]

[0020] Among them, I 0 The horizontal coordinate of Its value range is 1,2,3,…,W; the vertical coordinate is Its value range is 1, 2, 3, ..., H; V 0 The horizontal coordinate of Its value range is also 1, 2, 3, ..., W; the vertical coordinate is Its value range is also 1, 2, 3, …, H;

[0021] According to formula (4), the relative coordinates of visible light are calculated, thereby obtaining the visible light i vis and infrared registration image i ir .

[0022] Furthermore, the specific process of step 3, pre-training the fusion network is:

[0023] The formula is as follows:

[0024] Step 301, fusion network pre-training;

[0025] The fusion network pre-training initialization process uses an average strategy to pre-train the fusion model. This learning process can be expressed as:

[0026]

[0027] Where N F is a fusion network with learnable parameters θ, I ir and I vis represent infrared image and visible light image respectively, L WS is the fusion loss function, H and W represent the height and width of the source image respectively, I f represents the fused image, ||.||1 represents the calculation of L1 norm;

[0028] Step 302: semantic fusion network training;

[0029] By jointly training and fine-tuning the fusion network with the segmentation network, it can be expressed as:

[0030]

[0031] L ST =L sem +λL reg (8)

[0032] L reg =1 / (Corr(I ir ,I f )+Corr(I vis ,I f )) (9)

[0033] Where N S is a semantic segmentation network with learnable parameters ω, L ST is the semantic training loss function, L sem is the commonly used cross entropy loss function, L regis a regularization term, λ is used to balance the weights of these two terms, and Corr(·) represents the calculation of the correlation between the two image tensors.

[0034] Furthermore, the fusion network consists of a multi-dimensional feature extraction module and a lightweight multi-source perception fusion module based on a spatial attention mechanism; the multi-dimensional feature extraction module includes a multi-level fusion and extraction mechanism module and a multi-scale feature fusion and extraction mechanism module.

[0035] Furthermore, the multi-level fusion and extraction mechanism module is designed to extract the infrared image i from the input ir Or visible light image i vis The initial features are extracted from the image, and the features are extracted and fused step by step through multi-layer convolution operations. The input of the module is the infrared image i ir Or visible light image i vis , after preliminary convolution, the feature map is generated or As shown in formulas (10) and (11):

[0036]

[0037] Among them, Conv(·) represents 3*3 convolution.

[0038] Furthermore, the multi-scale feature fusion and extraction mechanism module generates feature maps of different resolutions through downsampling operations to extract multi-scale feature information, thereby enhancing the feature expression capability; the output of the multi-level fusion and extraction mechanism module and Through two downsampling operations and multi-level fusion and extraction mechanism modules, feature maps of two resolutions are obtained: and The downsampling formula is defined as follows:

[0039]

[0040] Among them, F ml (·) represents the above-mentioned multi-level fusion and extraction mechanism operation, and AvgPooling(·) represents the average pooling downsampling operation.

[0041] Furthermore, the input end of the lightweight multi-source perception fusion module based on the spatial attention mechanism is connected to the output end of the multi-dimensional feature extraction module; first, and Get the feature data generated by the maximum pooling and mean pooling, then concatenate these reduced-dimensional features, and then use a convolutional layer and activation layer to further learn and optimize this feature map; finally, get the infrared feature data Visible light characteristic data Perform weighted processing to obtain the fusion feature x fusion1 , expressed by formula (21):

[0042]

[0043] Among them, AvgPooling(·) represents the average pooling operation, MaxPooling(·) represents the maximum pooling operation, Conv(·) represents 3*3 convolution, concat(·) represents feature map concatenation, and α is a learnable parameter;

[0044] Similarly, the multi-dimensional feature extraction module is input into the spatial attention mechanism module to obtain three fused feature maps with different resolutions;

[0045] The output end of the lightweight multi-source perception fusion module based on the spatial attention mechanism undergoes multiple upsampling and convolution operations to output a fused image I fusion , where the upsampling formula is shown in formula (26):

[0046] x l =TConv(x l-1 ) (26)

[0047] Among them, TConv(·) is the deconvolution upsampling operation, x l and x l-1 They are the feature map outputs of layer l and layer l-1 respectively.

[0048] Furthermore, the semantic segmentation network is a SegFormer model, whose input is the fused image I fusion , the output is the segmented image I seg and the category and location of the target Obj;

[0049] (I seg ,Obj)=F seg (I fusion ) (27)

[0050] Among them, F seg It is the semantic segmentation model SegFormer.

[0051] The advantages of the present invention are: the present invention provides such a target segmentation and recognition method based on the fusion of visible light and infrared modalities, develops a semantic-level fusion network, does not rely on experimentally designed fusion rules and loss functions, and makes full use of semantic information to guide image fusion. Therefore, the fusion network can not only process and fuse multi-source image information, but also effectively understand and process the semantic content of the image while retaining important texture details. Compared with previous methods based on deep learning, this method not only reduces the computational pressure of the multi-source perception fusion module, but also improves the efficiency of dual-light image fusion based on the semantic fusion network. Moreover, it can break away from the limitations of the loss function designed by humans, significantly improve the performance of advanced visual tasks while ensuring high-quality visual effects, and at the same time improve the application value and effect of image fusion technology.

[0052] The present invention is described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of the target segmentation and recognition method based on the fusion of visible light and infrared modalities.

[0054] Figure 2 (a) is a schematic diagram of the original image.

[0055] Figure 2 (b) is a schematic diagram of the infrared image.

[0056] Figure 2 (c) is a schematic diagram of a visible light image.

[0057] Figure 2 (d) is a schematic diagram of the fused image. DETAILED DESCRIPTION

[0058] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the specific implementation methods, structural features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and embodiments.

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0061] Any feature disclosed in this specification (including any additional claims, abstract and drawings), unless otherwise stated, may be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0062] Example 1

[0063] This embodiment proposes a Figure 1 The target segmentation and recognition method based on visible light and infrared modality fusion shown includes the following steps:

[0064] Step 1: Get the infrared image and record it as i i , the visible light image is recorded as i v ; Infrared images and visible light images can be collected by corresponding sensors;

[0065] Step 2: The infrared image is recorded as i i , the visible light image is recorded as i v Perform registration preprocessing to obtain the infrared registration image, denoted as i ir , the visible light registration image is recorded as i vis ;

[0066] Step 3: Pre-train the fusion network;

[0067] Step 4: record the infrared registration image as i ir , the visible light registration image is recorded as i vis Input fusion network for fusion, output fusion image i f ;

[0068] Step 5: fusion image i f Perform semantic segmentation and identify and locate the target Obj.

[0069] Furthermore, in step 2, the infrared image is recorded as i i , the visible light image is recorded as i v Perform registration preprocessing to obtain the infrared registration image, denoted as i ir , the visible light registration image is recorded as i vis The specific process is:

[0070] Step 201, calculating the image relative parameter transformation matrix;

[0071] For infrared images, denoted as i i , the visible light image is denoted as i v Find N pairs of feature matching points from two multi-source images, and the infrared image is denoted as i i , the visible light image is recorded as i v The relative parameter transformation matrix P between I→VCalculated using the least squares formula:

[0072] P I→V =V*I T *(I*I T ) -1 (1)

[0073]

[0074] Where I represents the coordinates of the matching points of the infrared image as a 3×N matrix, V represents the coordinates of the matching points of the visible light image as a 3×N matrix, and I T is the transposed matrix of I, the relative parameter transformation matrix P I→V is a 3×3 matrix;

[0075] Step 202, transforming the visible light into the infrared image coordinate system;

[0076] According to the infrared image i Midpoint I 0 The coordinates of the mid-infrared image i are calculated i The coordinates of the corresponding point V 0 , and its calculation formula is:

[0077]

[0078] Among them, I 0 The horizontal coordinate of Its value range is 1,2,3,…,W; the vertical coordinate is Its value range is 1, 2, 3, ..., H; V 0 The horizontal coordinate of Its value range is also 1, 2, 3, ..., W; the vertical coordinate is Its value range is also 1, 2, 3, …, H;

[0079] According to formula (4), the relative coordinates of visible light are calculated, thereby obtaining the visible light i vis and infrared registration image i ir .

[0080] Furthermore, the specific process of step 3, pre-training the fusion network is:

[0081] The formula is as follows:

[0082] Step 301, fusion network pre-training;

[0083] The fusion network pre-training initialization process uses an average strategy to pre-train the fusion model. This learning process can be expressed as:

[0084]

[0085] Where N F is a fusion network with learnable parameters θ, I ir and I vis Respectively represent the infrared image and visible light image after registration, L WS is the fusion loss function, H and W represent the height and width of the source image respectively, I f represents the fused image, ||.||1 represents the calculation of L1 norm;

[0086] Step 302: semantic fusion network training;

[0087] By jointly training the fine-tuned fusion network with the Segformer segmentation network, it can be expressed as:

[0088]

[0089] L ST =L sem +λL reg (8)

[0090] L reg =1 / (Corr(I ir ,I f )+Corr(I vis ,I f )) (9)

[0091] Where N S is a semantic segmentation network with learnable parameters ω, L ST is the semantic training loss function, L sem is the commonly used cross entropy loss function, L reg is a regularization term, λ is used to balance the weights of these two terms, and Corr(·) represents the calculation of the correlation between the two image tensors.

[0092] Furthermore, the fusion network is composed of a multi-dimensional feature extraction module and a lightweight multi-source perception fusion module based on a spatial attention mechanism. The multi-dimensional feature extraction module implements multi-dimensional feature fusion and extraction from two aspects: feature level and feature scale. Figure 1 The multi-level fusion and extraction mechanism module, the multi-scale feature fusion and extraction mechanism module and the lightweight multi-source perception fusion module based on the spatial attention mechanism are described in detail.

[0093] The multi-level fusion and extraction mechanism module aims to obtain the infrared image i from the input ir Or visible light image i vis The module extracts preliminary features and realizes the step-by-step extraction and fusion of features through multi-layer convolution operations. The input of the module is the infrared image i ir Or visible light image ivis , after preliminary convolution, the feature map is generated or As shown in formulas (10) and (11):

[0094]

[0095] Among them, Conv(·) represents 3*3 convolution;

[0096] Depend on Figure 2 As shown in the figure, the multi-level fusion and extraction mechanism draws on the design mechanism of the Dense Block neural network structure to perform channel-dimensional splicing on the three convolutional feature maps, thereby achieving more efficient information flow and feature reuse. The output of the multi-level fusion and extraction mechanism module in the figure is recorded as and in:

[0097]

[0098] Among them, Conv(·) represents 3*3 convolution, concat(·) represents feature map concatenation, The feature maps generated by the first convolution and the second convolution are shown in equations (14) and (16), respectively. The feature maps generated for the first convolution and the second convolution are shown in equations (15) and (17) respectively.

[0099] The multi-scale feature fusion and extraction mechanism module generates feature maps of different resolutions through downsampling operations to extract multi-scale feature information, thereby enhancing feature expression capabilities. Figure 1 As shown, the output of the multi-level fusion and extraction mechanism module and Through two downsampling operations and multi-level fusion and extraction mechanism modules, feature maps of two resolutions are obtained: and The downsampling formula is defined as follows:

[0100]

[0101] Among them, F ml (·) represents the above-mentioned multi-level fusion and extraction mechanism operation, and AvgPooling(·) represents the average pooling downsampling operation.

[0102] Feature maps of three different levels and resolutions constitute a multi-dimensional feature extraction module.

[0103] The input end of the lightweight multi-source perception fusion module based on the spatial attention mechanism is connected to the output end of the multi-dimensional feature extraction module. Figure 1 As shown, first, and Get the feature data generated by the maximum pooling and mean pooling, then concatenate these reduced-dimensional features, and then use a convolutional layer and activation layer to further learn and optimize this feature map. Finally, get the infrared feature data Visible light characteristic data Perform weighted processing to obtain the fusion feature x fusion1 , expressed by formula (21):

[0104]

[0105]

[0106] Among them, AvgPooling(·) represents the average pooling operation, MaxPooling(·) represents the maximum pooling operation, Conv(·) represents 3*3 convolution, concat(·) represents feature map concatenation, and α is a learnable parameter.

[0107] Similarly, the multi-dimensional feature extraction module is input into the spatial attention mechanism module to obtain three fused feature maps of different resolutions.

[0108] The output end of the lightweight multi-source perception fusion module based on the spatial attention mechanism undergoes multiple upsampling and convolution operations to output a fused image I fusion , where the upsampling formula is shown in formula (26):

[0109] x l =TConv(x l-1 ) (26)

[0110] Among them, TConv(·) is the deconvolution upsampling operation, x l and x l-1 They are the feature map outputs of layer l and layer l-1 respectively.

[0111] The semantic segmentation network is a SegFormer model, whose input is the fused image i fusion , the output is the segmented image I seg and the category and location of the target Obj.

[0112] (I seg ,Obj)=F seg (I fusion ) (27)

[0113] Among them, F seg It is the semantic segmentation model SegFormer.

[0114] This embodiment proposes a multi-dimensional feature extraction module, which realizes multi-dimensional feature fusion and extraction from two aspects: feature level and feature scale.

[0115] First, a multi-level fusion and extraction mechanism is established. This module draws on the design mechanism of the Dense Block neural network structure and applies the dense connection strategy between each convolutional layer. Specifically, each layer not only receives the output of the previous layer, but also receives the output of higher-level features. Therefore, the features are not simply superimposed, but spliced ​​through the channel dimension of the feature map, thereby achieving more efficient information flow and feature reuse. This approach ensures gradual feature enhancement from shallow to deep layers, enabling the network to effectively process and fuse features from different levels while maintaining low computational complexity.

[0116] Secondly, a multi-scale feature fusion and extraction mechanism is established, which is used to extract feature information of different scales in visible light and infrared data in parallel, including texture details, shape features, high-order semantics, etc. Specifically, feature maps of different resolutions are obtained by downsampling. Among them, low-scale features contain more texture information and shape features, which are important factors affecting the visual effect of the image; while high-scale feature maps contain more semantic information, which is very important for understanding the overall content and context of the image, and plays a key role in the subsequent high-level visual task processing. Low-scale texture information and shape features can be directly integrated into the high-scale semantic information extraction process, enhancing the transmission efficiency and reusability of features. Multi-scale features not only improve the ability to capture details and shape features, but also strengthen the understanding of high-scale semantics. This structure has a direct positive impact on the visual effect of the image, especially when processing the fusion of visible light and infrared images in complex scenes, it can perform feature parsing and information extraction more accurately.

[0117] In order to fuse the extracted cross-modal feature maps, this embodiment proposes a lightweight multi-source perception fusion module based on the spatial attention mechanism. The core of the multi-source perception fusion module at each scale is to use the spatial attention mechanism to dynamically adjust and optimize the contribution of different data sources in the fusion process, thereby improving the overall performance and efficiency of the system.

[0118] This module obtains the perception data streams from the infrared feature channel and the visible light feature channel, and intelligently weights each data source on the corresponding channel according to its relevance and importance in the specific task or environmental context. Specifically: First, the feature data generated by the multi-dimensional feature extraction module is obtained. For each feature channel, a dimensionality reduction operation is used to extract the main information in the channel, including the maximum pooling and mean pooling results. Then, these reduced-dimensional features are concatenated to form a comprehensive feature map. A convolutional layer is used to further learn and optimize this feature map. Finally, the convolution output result is obtained to weight the feature data generated by the multi-dimensional feature extraction module, highlighting the important feature areas that are effective for fusion.

[0119] In summary, the target segmentation and recognition method based on the fusion of visible light and infrared modalities provided in this embodiment develops a semantic-level fusion network that does not rely on experimentally designed fusion rules and loss functions, and makes full use of semantic information to guide image fusion. Therefore, the fusion network can not only process and fuse multi-source image information, but also effectively understand and process the semantic content of the image while retaining important texture details. Compared with previous methods based on deep learning, this method not only reduces the computational pressure of the multi-source perception fusion module, but also improves the efficiency of dual-light image fusion based on the semantic fusion network. Moreover, it can break away from the limitations of the loss function designed by humans, significantly improve the performance of advanced visual tasks while ensuring high-quality visual effects, and at the same time improve the application value and effect of image fusion technology.

[0120] Example 2

[0121] Using the target segmentation and recognition method based on visible light and infrared modality fusion shown in Example 1, Figure 2 The images of the evening, morning and evening shown in (a) are processed. Figure 2 (b) is a schematic diagram of infrared images at night, morning, and dusk; Figure 2 (c) is a schematic diagram of visible light images at night, morning, and dusk; the developed fusion network can not only process and fuse multi-source image information, but also effectively understand and process the semantic content of the image while retaining important texture details. Figure 2 (d) It can be seen that the present invention has a very good fusion effect on dual-light images at night, morning and dusk. The fused image not only has the infrared contour and temperature information, but also has the detailed features such as the texture and color of visible light, and effectively understands and processes the semantic content of the image. It can also be seen from the figure that the present invention can accurately segment pedestrians and vehicle targets.

[0122] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A target segmentation and recognition method based on visible light and infrared modality fusion, characterized in that: The steps include: Step 1: Get the infrared image and record it as i i , the visible light image is recorded as i v ; Infrared images and visible light images can be collected by corresponding sensors; Step 2: The infrared image is recorded as i i , the visible light image is recorded as i v Perform registration preprocessing to obtain the infrared registration image, denoted as i ir , the visible light registration image is recorded as i vis ; Step 3: Pre-train the fusion network; Step 4: record the infrared registration image as i ir , the visible light registration image is recorded as i vis Input fusion network for fusion, output fusion image i f ; Step 5: fusion image i f Perform semantic segmentation and identify and locate the target Obj.

2. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 1, characterized in that: In step 2, the infrared image is recorded as i i , the visible light image is recorded as i v Perform registration preprocessing to obtain the infrared registration image, denoted as i ir , the visible light registration image is recorded as i vis The specific process is: Step 201, calculating the image relative parameter transformation matrix; For infrared images, denoted as i i , the visible light image is denoted as i v Find N pairs of feature matching points from two multi-source images, and the infrared image is denoted as i i , the visible light image is recorded as i v The relative parameter transformation matrix PI→V between is calculated using the least squares formula: PI→V=V*I T *(I*I T )- 1 (1) Where I represents the coordinates of the matching points of the infrared image as a 3×N matrix, V represents the coordinates of the matching points of the visible light image as a 3×N matrix, and I T is the transposed matrix of I, the relative parameter transformation matrix P I→V is a 3×3 matrix; Step 202, transforming the visible light into the infrared image coordinate system; According to the infrared image i Midpoint I 0 The coordinates of the reference image i are calculated i The coordinates of the corresponding point V 0 , and its calculation formula is: Among them, I 0 The horizontal coordinate of Its value range is 1, 2, 3, ..., W; the vertical coordinate is Its value range is 1, 2, 3, ..., H; V 0 The horizontal coordinate of Its value range is also 1, 2, 3, ..., W; the vertical coordinate is Its value range is also 1, 2, 3, ..., H; According to formula (4), the relative coordinates of visible light are calculated, thereby obtaining the visible light i vis and infrared registration image i ir .

3. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 1, characterized in that: The specific process of step 3, pre-training the fusion network is as follows: Step 301, fusion network pre-training; The fusion network pre-training initialization process uses an average strategy to pre-train the fusion model. This learning process can be expressed as: Where N F is a fusion network with learnable parameters θ, I ir and I vis represent infrared image and visible light image respectively, L WS is the fusion loss function, H and W represent the height and width of the source image respectively, I f represents the fused image, ||.||1 represents the calculation of L1 norm; Step 302: semantic fusion network training; By jointly training and fine-tuning the fusion network with the segmentation network, it can be expressed as: THE ST =L sem +λL reg (8) L reg =1 / (Corr(I ir ,I f )+Corr(I vis ,I f )) (9) Where N S is a semantic segmentation network with learnable parameters ω, L ST is the semantic training loss function, L sem is the commonly used cross entropy loss function, L reg is a regularization term, λ is used to balance the weights of these two terms, and Corr(·) represents the calculation of the correlation between the two image tensors.

4. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 1, characterized in that: The fusion network consists of a multi-dimensional feature extraction module and a lightweight multi-source perception fusion module based on a spatial attention mechanism; the multi-dimensional feature extraction module includes a multi-level fusion and extraction mechanism module and a multi-scale feature fusion and extraction mechanism module.

5. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 4, characterized in that: The multi-level fusion and extraction mechanism module aims to obtain the infrared image i from the input ir Or visible light image i vis The initial features are extracted from the image, and the features are extracted and fused step by step through multi-layer convolution operations. The input of the module is the infrared image i ir Or visible light image i vis , after preliminary convolution, the feature map is generated or As shown in formulas (10) and (11): Among them, Conv(·) represents 3*3 convolution.

6. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 4, characterized in that: The multi-scale feature fusion and extraction mechanism module generates feature maps of different resolutions through downsampling operations to extract multi-scale feature information, thereby enhancing the feature expression capability; the output of the multi-level fusion and extraction mechanism module and Through two downsampling operations and multi-level fusion and extraction mechanism modules, feature maps of two resolutions are obtained: and The downsampling formula is defined as follows: Among them, F ml (·) represents the above-mentioned multi-level fusion and extraction mechanism operation, and AvgPooling(·) represents the average pooling downsampling operation.

7. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 4, characterized in that: The input end of the lightweight multi-source perception fusion module based on the spatial attention mechanism is connected to the output end of the multi-dimensional feature extraction module; first, and Get the feature data generated by the maximum pooling and mean pooling, then concatenate these reduced-dimensional features, and then use a convolutional layer and activation layer to further learn and optimize this feature map; finally, get the infrared feature data Visible light characteristic data Perform weighted processing to obtain the fusion feature x fusion1 , expressed by formula (21): Among them, AvgPooling(·) represents the average pooling operation, MaxPooling(·) represents the maximum pooling operation, Conv(·) represents 3*3 convolution, concat(·) represents feature map concatenation, and α is a learnable parameter; Similarly, the multi-dimensional feature extraction module is input into the spatial attention mechanism module to obtain three fused feature maps with different resolutions; The output end of the lightweight multi-source perception fusion module based on the spatial attention mechanism undergoes multiple upsampling and convolution operations to output a fused image I fusion , where the upsampling formula is shown in formula (26): x l =TConv(x l-1 ) (26) Among them, TConv(·) is the deconvolution upsampling operation, x l and x l-1 They are the feature map outputs of layer l and layer l-1 respectively.

8. The target segmentation and recognition method based on visible light and infrared modality fusion as claimed in claim 3, characterized in that: The semantic segmentation network is a SegFormer model, whose input is the fused image I fusion , the output is the segmented image I seg and the category and location of the target Obj; (I seg ,Obj)=F seg (I fusion ) (27) Among them, F seg It is the semantic segmentation model SegFormer.

Images