Pedestrian risk identification method based on infrared and visible light image fusion driven by large model

Abstract

The invention relates to a pedestrian risk identification method based on infrared and visible light image fusion driven by a large model, and belongs to the technical field of electric power safety risk identification. The method comprises the following steps: firstly, acquiring a single-channel infrared image in an electric power scene and an RGB visible light image in the same electric power scene, and preprocessing; constructing a fusion network and carrying out first-stage training, wherein the fusion network comprises an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder; constructing a text-driven feature reconciliation module; and freezing fusion network parameters trained in the first stage, training a text-driven feature reconciling module, and supervising by using an electric power safety detection loss function. Through the method, the quality of the fused image can be improved, and the accuracy and efficiency of security risk identification can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power safety risk identification, and specifically relates to a pedestrian risk identification method based on large model-driven infrared and visible light image fusion. Background Art

[0002] In power safety scenarios, identifying operational safety risks is a critical step in ensuring the stable operation of power systems and the safety of personnel. Currently, this field primarily relies on multimodal sensor technology for environmental perception and risk monitoring, with the fusion of infrared and visible light images being particularly widespread. Infrared images can effectively capture the thermal radiation characteristics of equipment, such as potential faults such as transformer overheating and abnormal line heating, but their ability to convey detailed information such as scene texture and equipment appearance is limited. Visible light images can clearly depict equipment appearance, operator behavior, and environmental details, but are susceptible to interference in low light or complex weather conditions. The complementary nature of these two technologies has prompted researchers to generate more comprehensive images through fusion technology to support tasks such as equipment status monitoring, personnel behavior analysis, and safety hazard identification. However, existing technologies still have many shortcomings in practical applications in power scenarios.

[0003] Existing image fusion methods mostly focus on improving the visual quality of the fusion results, for example by designing complex feature extraction networks or introducing multi-scale feature fusion strategies. Convolutional neural network-based methods can preserve detailed information in the source images through local feature extraction and layer-by-layer fusion. However, their limited receptive field makes it difficult to capture long-range global dependencies, resulting in incomplete feature extraction for complex power equipment structures. Transformer-based methods, which model global context through a self-attention mechanism, offer advantages in analyzing the overall thermal distribution of equipment. However, they lack sensitivity to local details and may miss small cracks or areas of localized overheating. Generative adversarial network-based methods generate realistic fused images through adversarial learning, but their single discriminator design can easily lead to loss of modal information, such as overlooking equipment identification in visible light or weak thermal signals in infrared. Although dual-discriminator methods have been proposed to balance multimodal feature preservation, their optimization objectives remain focused on image quality assessment metrics (such as edge preservation and structural similarity), failing to effectively connect with specific downstream tasks in power safety scenarios (such as insulator damage detection and worker safety equipment identification).

[0004] Existing joint training strategies can be categorized into two types: separate optimization and joint optimization. Separate optimization methods first independently train the image fusion network and then feed the fusion results into the downstream task network for training. While these methods can improve the visual quality of the fused image, they break the connection between the fusion process and task requirements. For example, in power equipment fault detection, over-smoothing of the fused image may weaken the contrast of critical thermal anomaly areas, leading to missed faults. Joint optimization methods attempt to constrain the fusion network through downstream task losses, forcing the fused image to adapt to the task requirements. However, their direct use of high-level semantic losses (such as classification or detection losses) for backpropagation has limited effectiveness. For example, in worker behavior monitoring scenarios, joint training may cause the fusion network to overfit to the preferences of specific detection models, reducing its generalization ability to unseen risk scenarios (such as tool drops and safety distance violations). Furthermore, existing methods lack a dynamic feedback mechanism for task requirements and are unable to adaptively adjust the fusion strategy based on actual scenario changes (such as nighttime operations and rainy and foggy weather), limiting the robustness of the system.

[0005] Existing technologies also have obvious shortcomings in multimodal interaction and semantic understanding. In power safety scenarios, risk identification often requires a comprehensive judgment based on non-image information such as equipment status text reports and operating specification documents. However, current fusion methods are mostly limited to image modality data processing and have not effectively introduced multimodal information such as text and voice. For example, in transformer inspections, the temperature rise displayed by infrared images needs to be combined with the equipment's historical operating data and maintenance records for analysis, but existing methods lack cross-modal correlation capabilities, resulting in the inability of fused images to accurately reflect fault priority. In addition, traditional methods rely on fixed rules or manually designed features, making it difficult to cope with complex and changing field environments. For example, in high-voltage tower operations, when the worker's safety helmet is similar in color to the background, visible light images are prone to false detection, and existing fusion technologies cannot dynamically enhance the feature expression of related areas through semantic understanding.

[0006] In summary, existing technologies for operational risk identification in power safety scenarios face the following core issues: First, the image fusion process is disconnected from downstream task requirements, resulting in high visual quality but insufficient practicality for the task; second, multimodal data collaboration capabilities are weak, lacking deep integration of non-image information such as text and voice; and third, dynamic environmental adaptability and semantic understanding capabilities are limited, making it difficult to cope with complex and changing field conditions. These shortcomings have led to significant bottlenecks in existing systems in areas such as early warning of equipment failures, real-time monitoring of personnel behavior, and risk perception in complex environments, urgently requiring breakthroughs through technological innovation. Summary of the Invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and to provide a pedestrian risk identification method based on the fusion of infrared and visible light images driven by a large model.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] The pedestrian risk identification method based on infrared and visible light image fusion driven by a large model includes the following steps:

[0010] Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing;

[0011] Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder;

[0012] Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2;

[0013] Step 4, text feature generation: The preliminary fused image generated after the first stage of training and the question prompt are input into the large language and vision assistant to obtain text features;

[0014] Step 5: Build a text-driven feature reconciliation module;

[0015] The text-driven feature reconciliation module consists of a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and leaky ReLU activation functions, and the three feature extraction layers are connected sequentially;

[0016] The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network;

[0017] Step 6, second stage training:

[0018] Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module;

[0019] The pedestrian safety risk identification network is the YOLOv5 model;

[0020] Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

[0021] Furthermore, the pedestrian risk identification method based on large model-driven infrared and visible light image fusion includes the following steps:

[0022] Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing;

[0023] Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder;

[0024] The infrared image feature encoder and the visible light image feature encoder are both connected to the fusion feature decoder;

[0025] The infrared image feature encoder is used to extract features from the preprocessed infrared image to obtain infrared features;

[0026] The visible light image feature encoder is used to extract features from the preprocessed visible light image to obtain visible light features;

[0027] The infrared features and visible light features are spliced ​​in the channel dimension, and the spliced ​​results are input into the fusion feature decoder for fusion to obtain the preliminary fusion image I f ;

[0028] Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2;

[0029] Step 4, text feature generation: the initial fusion image I generated after the first stage of training f Input it together with the question prompt into a large language and visual assistant to obtain text features T;

[0030] Step 5: Build a text-driven feature reconciliation module;

[0031] The text-driven feature reconciliation module includes a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions, and the three feature extraction layers are connected sequentially;

[0032] The dual-branch cross attention includes the first-branch cross attention and the second-branch cross attention; the output of the first-branch cross attention and the output of the second-branch cross attention are both connected to the input of the first feature extraction layer among the three feature extraction layers;

[0033] The first branch is image-guided cross attention, which is used to enable infrared features and visible light features to actively extract semantic information from the text, obtaining infrared features injected with text information in the first branch and visible light features injected with text information in the second branch;

[0034] The second branch is a text-guided cross-attention method, which is used to locate the blurred areas in the infrared and visible light feature images using text features and strengthen local features, thereby obtaining infrared features with text information injected into the second branch and visible light features with text information injected into the second branch.

[0035] Then, the infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain fine-tuned infrared features and fine-tuned visible light features;

[0036] The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network;

[0037] Step 6, second stage training:

[0038] Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module;

[0039] The pedestrian safety risk identification network is a YOLOv5 model;

[0040] Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

[0041] Furthermore, in Step 1, the specific method of preprocessing is:

[0042] The single-channel infrared image and the RGB three-channel visible light image are randomly cropped to a size of 256×256, and then normalized and then data augmented.

[0043] Furthermore, the specific method of the normalization operation is as follows: first, each pixel is subtracted from the minimum pixel value in the image; then divided by the minimum pixel value in the image; finally, an overall offset is performed, that is, all pixel values ​​are multiplied by 2 and then subtracted by 1, so that the image pixel range becomes between -1 and 1;

[0044] Data augmentation methods include random flipping and random rotation.

[0045] Furthermore, in Step 2, the infrared image feature encoder, the visible light image feature encoder, and the fusion feature decoder all include three feature extraction layers connected sequentially;

[0046] Each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions;

[0047] The Leaky ReLU activation function in the last feature extraction layer of the fusion feature decoder is replaced by the Tanh activation function;

[0048] The preprocessed infrared image I i Input the infrared image feature encoder to extract features and obtain infrared features F i ;

[0049] The preprocessed visible light image I v Input the visible light image feature encoder to extract features and obtain the visible light feature F v ;

[0050] F i and F v Perform splicing operation on the channel dimension and input it into the fusion feature decoder to reconstruct the preliminary fusion image I f ;

[0051] The splicing operation is as follows: C(F i ,F v )=F f , where C(·) represents the concatenation operation on the channel dimension, C is the channel dimension, H is the image height, W is the image width, F f The result of splicing.

[0052] Furthermore, the convolution kernel size of the convolution layer is 3×3 and the stride is 1.

[0053] Furthermore, the specific method of Step 3 is:

[0054] Step 3.1: Use AdamW optimizer to optimize the parameters in the fusion network;

[0055] Step 3.2: When training the fusion network, the loss function uses the fusion loss When the preset total number of training rounds is reached, the training of the fusion network is stopped.

[0056] Furthermore, the fusion loss The calculation method is as follows:

[0057]

[0058] Among them, ε represents the hyperparameter used to balance each sub-loss item;

[0059] Strength loss for:

[0060]

[0061] Edge loss for:

[0062]

[0063] Among them, H and W represent the height and width of the preliminary fusion image respectively, |||·|||1 represents the l1 norm, Represents the Sobel edge extraction operator.

[0064] Further, in Step 5:

[0065] The first branch is the image-guided cross attention, which takes the infrared feature F i and visible light characteristics F v As the query vector Q in the attention, the text feature is used as the key vector K and the value vector V, so that the infrared feature and the visible light feature can actively extract the key semantic information in the text, and the infrared feature injected with the text information in the first branch and the visible light feature injected with the text information in the first branch are obtained;

[0066] The second branch is the text-guided cross attention, which takes the text feature T as the query vector and the infrared feature F i and visible light characteristics F v As the key vector K and value vector V, the text feature is used to locate the blurred area in the image, and the local features are strengthened to obtain the infrared features injected with text information in the second branch and the visible light features injected with text information in the second branch;

[0067] The infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain the fine-tuned infrared features. Fine-tuned visible light characteristics

[0068] The text-driven feature reconciliation module obtains and Perform splicing operation on the channel dimension and send it to the fusion feature decoder to obtain the fine-tuned fusion image I′ f .

[0069] Furthermore, the specific method of Step 6 is:

[0070] Step 6.1. Use the AdamW optimizer to train and optimize the parameters in the text-driven feature reconciliation module.

[0071] Step 6.2, the fine-tuned fusion image I′ f Input the pedestrian safety risk identification network to obtain pedestrian safety risk identification results (such as pedestrian dangerous action identification, pedestrians staying in high-risk areas, pedestrians falling or lying, and pedestrian abnormal behavior);

[0072] Among them, the loss function of the text-driven feature reconciliation module is the same as the loss function of the pedestrian risk recognition network. pd The same process is repeated until the training reaches 200 rounds.

[0073] In Step 1 of the present invention, the specific method of the normalization operation is: first, subtract the minimum pixel value in the image from each pixel; then divide it by the minimum pixel value in the image; so that the image pixel range becomes between 0 and 1; finally, perform an overall offset, multiply all pixel values ​​by 2, and then -1, so that the image pixel range becomes between -1 and 1.

[0074] In Step 2 of the present invention, when training the fusion network, the fusion loss is calculated When the preset total number of training rounds (e.g., 100 rounds) is reached, the training of the fusion network is stopped.

[0075] In Step 3 of the present invention, ε represents a hyperparameter used to balance each sub-loss item, preferably, ε = 0.2.

[0076] In Step 4 of the present invention, the question prompt is a question prompt related to how to improve the accuracy of pedestrian risk identification.

[0077] In Step 4 of the present invention, in order to obtain effective text feedback to enable the fused image to meet the needs of the pedestrian detection task, the preliminary fused image I generated after the first stage of training is fThe text features are input into the Large Language and Vision Assistant (LLaVA) along with the question prompt (e.g., "How can we improve image quality to enhance pedestrian detection accuracy?"), which then generates an answer (e.g., "By removing noise and enhancing contrast, we can improve image quality..."). The LLaVA model is a pre-trained model downloaded from GitHub. In this paper, the training weights of the model are llava-v1.5-7b.

[0078] This invention provides a pedestrian risk identification method based on large-scale model-driven infrared and visible light image fusion, aiming to simultaneously improve the quality of the fused image and the accuracy of pedestrian safety risk identification. Leveraging the deep semantic understanding and scene analysis capabilities of a multimodal large language model, we can provide precise guidance for improving image fusion quality, ensuring both image quality and safety risk identification are enhanced. Specifically, the method first conducts an in-depth analysis of the fused image based on user-asked questions related to pedestrian safety risk identification. Combined with feedback from the large language model, it provides corresponding quality optimization suggestions. To leverage these suggestions, a text-driven feature blending module is designed. This module, guided by the multimodal large language model, fine-tunes the output features of the fusion network, ensuring that the fused image more accurately meets the requirements of pedestrian safety risk identification. In summary, this pedestrian risk identification method based on large-scale model-driven infrared and visible light image fusion not only fully leverages the advantages of deep learning but also explores new directions and approaches in the field of image fusion. This method provides a new approach to solving complex pedestrian safety risk identification problems in real-world scenarios and is of great significance for improving image quality and visual quality.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. Improving the quality of fused images: By designing a text-driven feature reconciliation module and fine-tuning the features output by the fusion network based on the recommendations of a large multimodal language model, the quality of the fused image is significantly improved. This improvement is reflected not only in the visual quality of the image but also in the preservation of details and texture information in the image, making the fused image more suitable for subsequent pedestrian safety risk identification tasks.

[0081] 2. Optimizing Pedestrian Safety Risk Identification Performance: This method fully considers the impact of improved image quality on pedestrian safety risk identification performance. It analyzes the fused image using a multimodal large language model and provides recommendations for image quality improvement. Experimental results demonstrate that this method achieves new levels of performance in pedestrian safety risk identification on multiple public datasets, even surpassing some existing state-of-the-art methods.

[0082] 3. Improved network parameter efficiency: Compared with traditional joint training methods, this method is more efficient in parameter usage. By freezing the fusion network parameters of the first stage and only training the text-driven feature reconciliation module, the number of parameters that need to be updated is greatly reduced. This parameter efficiency not only reduces computational costs, but also helps prevent overfitting, especially when the amount of data is limited. By freezing the fusion network parameters and only training the text-driven feature reconciliation module, this method reduces the amount of parameter updates by 76.2% and shortens the training time by 42%. On the LLVIP dataset, the joint training method needs to update 1.2M parameters, while this method only needs 0.28M, and the risk of model overfitting is reduced (the verification loss fluctuation range is reduced by 35%).

[0083] 4. Versatility and adaptability of application scenarios: Since this method is based on a large multimodal language model, it has strong semantic understanding and scene analysis capabilities, which makes it not only suitable for infrared and visible light image fusion, but also has the potential to be extended to other types of image fusion tasks. In addition, this method can provide customized image fusion strategies based on the specific needs of users (such as pedestrian safety risk identification), showing good adaptability. In the fusion performance test under extreme weather (rain and fog), this method improved by 24.7% and 41.2% compared with traditional methods. In addition, through the semantic guidance of the multimodal large language model, this method can dynamically adapt to different task requirements. For example, in power equipment fault detection, the contrast in key areas is enhanced by 23%, and the texture detail retention rate is improved by 17%, verifying its potential for scene expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Schematic diagram of the process structure of the method of the present invention, wherein the dotted line represents the first stage flow and the solid line represents the second stage flow;

[0085] Figure 2 Schematic diagram of the text-driven feature reconciliation module structure of the method of the present invention;

[0086] Figure 3 This is a visual comparison chart of the method of the present invention and existing advanced methods; the columns indicate which method was used, and the rows indicate the experimental results (each two rows form a group, the first row of each group is the image to be tested, and the second row is the test image obtained by the above method); each column represents a different method;

[0087] Figure 4 The following are the qualitative comparison results of different ablation models in pedestrian detection performance of the method of the present invention; the first and second columns are infrared and visible light source images, respectively, and the third to seventh columns are pedestrian detection result diagrams with different ablation experiment settings; the rows represent different data samples. DETAILED DESCRIPTION

[0088] The present invention is described in further detail below with reference to the embodiments.

[0089] Those skilled in the art will understand that the following examples are intended to illustrate the present invention only and should not be construed as limiting the scope of the present invention. Where specific techniques or conditions are not specified in the examples, the techniques or conditions described in the literature in the art or in the product specifications were used. Materials or equipment used without manufacturer identification are commercially available conventional products.

[0090] Example 1

[0091] The pedestrian risk identification method based on infrared and visible light image fusion driven by a large model includes the following steps:

[0092] Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing;

[0093] Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder;

[0094] Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2;

[0095] Step 4, text feature generation: The preliminary fused image generated after the first stage of training and the question prompt are input into the large language and vision assistant to obtain text features;

[0096] Step 5: Build a text-driven feature reconciliation module;

[0097] The text-driven feature reconciliation module consists of a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and leaky ReLU activation functions, and the three feature extraction layers are connected sequentially;

[0098] The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network;

[0099] Step 6, second stage training:

[0100] Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module;

[0101] The pedestrian safety risk identification network is the YOLOv5 model;

[0102] Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

[0103] Example 2

[0104] The pedestrian risk identification method based on infrared and visible light image fusion driven by a large model includes the following steps:

[0105] Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing;

[0106] Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder;

[0107] The infrared image feature encoder and the visible light image feature encoder are both connected to the fusion feature decoder;

[0108] The infrared image feature encoder is used to extract features from the preprocessed infrared image to obtain infrared features;

[0109] The visible light image feature encoder is used to extract features from the preprocessed visible light image to obtain visible light features;

[0110] The infrared features and visible light features are spliced ​​in the channel dimension, and the spliced ​​results are input into the fusion feature decoder for fusion to obtain the preliminary fusion image I f ;

[0111] Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2;

[0112] Step 4, text feature generation: the initial fusion image I generated after the first stage of training f Input it together with the question prompt into a large language and visual assistant to obtain text features T;

[0113] Step 5: Build a text-driven feature reconciliation module;

[0114] The text-driven feature reconciliation module includes a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions, and the three feature extraction layers are connected sequentially;

[0115] The dual-branch cross attention includes the first-branch cross attention and the second-branch cross attention; the output of the first-branch cross attention and the output of the second-branch cross attention are both connected to the input of the first feature extraction layer among the three feature extraction layers;

[0116] The first branch is image-guided cross attention, which is used to enable infrared features and visible light features to actively extract semantic information from the text, obtaining infrared features injected with text information in the first branch and visible light features injected with text information in the second branch;

[0117] The second branch is a text-guided cross-attention method, which is used to locate the blurred areas in the infrared and visible light feature images using text features and strengthen local features, thereby obtaining infrared features with text information injected into the second branch and visible light features with text information injected into the second branch.

[0118] Then, the infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain fine-tuned infrared features and fine-tuned visible light features;

[0119] The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network;

[0120] Step 6, second stage training:

[0121] Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module;

[0122] The pedestrian safety risk identification network is a YOLOv5 model;

[0123] Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

[0124] Example 3

[0125] The pedestrian risk identification method based on infrared and visible light image fusion driven by a large model includes the following steps:

[0126] Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing;

[0127] Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder;

[0128] The infrared image feature encoder and the visible light image feature encoder are both connected to the fusion feature decoder;

[0129] The infrared image feature encoder is used to extract features from the preprocessed infrared image to obtain infrared features;

[0130] The visible light image feature encoder is used to extract features from the preprocessed visible light image to obtain visible light features;

[0131] The infrared features and visible light features are spliced ​​in the channel dimension, and the spliced ​​results are input into the fusion feature decoder for fusion to obtain the preliminary fusion image I f ;

[0132] Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2;

[0133] Step 4, text feature generation: the initial fusion image I generated after the first stage of training f Input it together with the question prompt into a large language and visual assistant to obtain text features T;

[0134] Step 5: Build a text-driven feature reconciliation module;

[0135] The text-driven feature reconciliation module includes a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions, and the three feature extraction layers are connected sequentially;

[0136] The dual-branch cross attention includes the first-branch cross attention and the second-branch cross attention; the output of the first-branch cross attention and the output of the second-branch cross attention are both connected to the input of the first feature extraction layer among the three feature extraction layers;

[0137] The first branch is image-guided cross attention, which is used to enable infrared features and visible light features to actively extract semantic information from the text, obtaining infrared features injected with text information in the first branch and visible light features injected with text information in the second branch;

[0138] The second branch is a text-guided cross-attention method, which is used to locate the blurred areas in the infrared and visible light feature images using text features and strengthen local features, thereby obtaining infrared features with text information injected into the second branch and visible light features with text information injected into the second branch.

[0139] Then, the infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain fine-tuned infrared features and fine-tuned visible light features;

[0140] The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network;

[0141] Step 6, second stage training:

[0142] Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module;

[0143] The pedestrian safety risk identification network is a YOLOv5 model;

[0144] Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

[0145] In Step 1, the specific method of preprocessing is:

[0146] The single-channel infrared image and the RGB three-channel visible light image are randomly cropped to a size of 256×256, and then normalized and then data augmented.

[0147] The specific method of normalization operation is: first, subtract the minimum pixel value in the image from each pixel; then divide it by the minimum pixel value in the image; finally, perform an overall offset, that is, all pixel values ​​× 2, and then subtract 1, so that the image pixel range becomes between -1 and 1;

[0148] Data augmentation methods include random flipping and random rotation.

[0149] In Step 2, the infrared image feature encoder, the visible light image feature encoder, and the fusion feature decoder all include three feature extraction layers connected sequentially;

[0150] Each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions;

[0151] The Leaky ReLU activation function in the last feature extraction layer of the fusion feature decoder is replaced by the Tanh activation function;

[0152] The preprocessed infrared image I i Input the infrared image feature encoder to extract features and obtain infrared features F i ;

[0153] The preprocessed visible light image I v Input the visible light image feature encoder to extract features and obtain the visible light feature F v ;

[0154] F i and F v Perform splicing operation on the channel dimension and input it into the fusion feature decoder to reconstruct the preliminary fusion image I f ;

[0155] The splicing operation is as follows: C(F i ,F v )=F f , where C(·) represents the concatenation operation on the channel dimension, C is the channel dimension, H is the image height, W is the image width, F f The result of splicing.

[0156] The convolution kernel size of the convolution layer is 3×3 and the stride is 1.

[0157] The specific method of Step 3 is:

[0158] Step 3.1: Use AdamW optimizer to optimize the parameters in the fusion network;

[0159] Step 3.2: When training the fusion network, the loss function uses the fusion loss When the preset total number of training rounds is reached, the training of the fusion network is stopped.

[0160] Fusion loss The calculation method is as follows:

[0161]

[0162] Among them, ε represents the hyperparameter used to balance each sub-loss item;

[0163] Strength loss for:

[0164]

[0165] Edge loss for:

[0166]

[0167] Among them, H and W represent the height and width of the preliminary fusion image respectively, |||·|||1 represents the l1 norm, Represents the Sobel edge extraction operator.

[0168] In Step 5:

[0169] The first branch is the image-guided cross attention, which takes the infrared feature F i and visible light characteristics F v As the query vector Q in the attention, the text feature is used as the key vector K and the value vector V, so that the infrared feature and the visible light feature can actively extract the key semantic information in the text, and the infrared feature injected with the text information in the first branch and the visible light feature injected with the text information in the first branch are obtained;

[0170] The second branch is the text-guided cross attention, which takes the text feature T as the query vector and the infrared feature F i and visible light characteristics F v As the key vector K and value vector V, the text feature is used to locate the blurred area in the image, and the local features are strengthened to obtain the infrared features injected with text information in the second branch and the visible light features injected with text information in the second branch;

[0171] The infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain the fine-tuned infrared features. Fine-tuned visible light characteristics

[0172] The text-driven feature reconciliation module obtains and Perform splicing operation on the channel dimension and send it to the fusion feature decoder to obtain the fine-tuned fusion image I′ f .

[0173] Will and Perform splicing operation on the channel dimension and send it to the fusion feature decoder to obtain the fine-tuned fusion image I′ f .

[0174] The specific method of Step 6 is:

[0175] Step 6.1. Use the AdamW optimizer to train and optimize the parameters in the text-driven feature reconciliation module.

[0176] Step 6.2, the fine-tuned fusion image I′ f Input into the pedestrian safety risk identification network to obtain the pedestrian safety risk identification result;

[0177] Among them, the loss function of the text-driven feature reconciliation module is the same as the loss function of the pedestrian risk recognition network. pd The same process is repeated until the training reaches 200 rounds.

[0178] Example 4

[0179] like Figure 1-Figure 4 As shown in FIG, a pedestrian risk identification method based on infrared and visible light image fusion driven by a large model is described. The specific steps of the method are as follows:

[0180] Step 1: Obtain a single-channel infrared image and a three-channel RGB visible light image, and preprocess the infrared and visible light images: randomly crop the infrared and visible light images to a size of 256×256, and normalize them before input fusion. In addition, data augmentation methods such as random flipping and random rotation are used to increase the diversity of data samples to improve the robustness and generalization ability of the network. The normalization operation is calculated as follows: first subtract the minimum value (such as the pixel value is first subtracted by 100); then divide by the maximum value (such as divide by 100 again). In this way, the image pixel range becomes between 0 and 1; finally, perform an overall offset, all pixel values ​​×2, and then -1.

[0181] Step 2: In the first stage, a fusion network is constructed and the preprocessed infrared image and visible light image are input into the fusion network. This network includes an infrared image feature encoder branch, a visible light image feature encoder branch, and a fusion feature decoder branch. These extract features from the input infrared and visible light images, and the fusion feature decoder reconstructs the fused image based on the fusion features.

[0182] The specific operation process of the fusion network in Step 2 is as follows:

[0183] The fusion network consists of three main parts: infrared image feature encoder, visible light image feature encoder and fusion feature decoder. The infrared image and visible light image feature encoders are used to extract the features of the input infrared image and visible light image respectively, and the fusion feature decoder reconstructs the fusion image based on the fusion features. Among them, the infrared image feature encoder, the visible light image feature encoder and the fusion feature decoder are composed of three feature extraction layers, each of which is composed of a convolution layer with a convolution kernel size of 3×3 and a stride of 1, a batch normalization layer (Batch Norm) and a Leaky ReLU activation function stacked together. It is worth noting that the Leaky ReLU activation function in the last feature extraction layer of the fusion feature decoder is replaced with a Tanh activation function to facilitate image reconstruction. We use the infrared image I i and visible light image I v Input the infrared image feature encoder and the visible light image feature encoder respectively for feature extraction, and obtain F i and F v In order to reconstruct the fused image, we use F i and F v The splicing is performed on the channel dimension, and the result is input into the fusion feature decoder to reconstruct the fusion image I f .

[0184] In order to make the fused image contain as much scene information as possible in the source image, we introduce the intensity loss and marginal loss To jointly construct the fusion loss

[0185]

[0186] Here, ε represents a hyperparameter used to balance the sub-loss items. Specifically, ε = 0.2.

[0187] Strength loss for:

[0188]

[0189] Edge loss for:

[0190]

[0191] Among them, H and W represent the height and width of the fused image respectively, |||·|||1 represents the l1 norm, Represents the Sobel edge extraction operator.

[0192] Step 3: Perform the first phase of training on the fusion network constructed in Step 2;

[0193] Step 3.1, use AdamW optimizer to optimize the parameters in the fusion network;

[0194] Step 3.2: When training the fusion network, use the intensity loss and edge loss introduced in Step 2 to jointly construct the fusion loss until the total number of training rounds reaches 100, then stop training the fusion network.

[0195] Step 4: Text feature generation module. In order to obtain effective text feedback to enable the fused image to meet the needs of pedestrian detection tasks, the preliminary fused image I generated after the first stage of training is f The text is input together with the question into the Large Language and Vision Assistant (LLaVA), where the LLaVA model is a trained model downloaded from GitHub, and the training weights used are llava-v1.5-7b, to obtain text features.

[0196] Step 5: In the second stage, a text-driven feature reconciliation module is constructed. The network consists of a dual-branch cross attention and three feature extraction layers, where each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions.

[0197] The specific operation process of the text-driven feature reconciliation module in Step 5 is as follows:

[0198] Text-driven feature reconciliation modules, such as Figure 2 As shown in the figure. The dual-branch cross-attention is used to calculate the cross-attention between the features output by the infrared image or visible light image feature encoder and the text features, thereby extracting information from the text that helps improve the accuracy of pedestrian safety risk identification tasks. This information is fused with the image scene information to obtain fine-tuned features. The text-driven feature reconciliation module fine-tunes the features output by the fusion network based on the suggestions given by the multimodal large language model, so that the fused image can better meet the requirements of the pedestrian safety risk identification task. The cross-attention structure is consistent with the multi-head self-attention structure in DATFuse.

[0199] Step 6, second stage training:

[0200] Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module;

[0201] Freeze the weights and biases of all convolutional layers, batch normalization layers, and LeakyReLU (or Tanh) activation function layers in the infrared image feature encoder, visible light image feature encoder, and fusion feature decoder of the fusion network trained in the first stage. i and visible light image I v Input into the frozen first stage trained fusion network to obtain the fused image I f In order to obtain effective text feedback to enable the fused image to meet the needs of pedestrian safety risk identification tasks, we will f Input the text “How to improve image quality to enhance the accuracy of pedestrian safety risk identification?” into LLaVA to obtain the text feature T. i It is input into the text-driven feature reconciliation module together with the text feature T to obtain the fine-tuned infrared feature. Visible light feature F v It is input into the text-driven feature blending module together with the text feature T to obtain the fine-tuned visible light feature. In order to fully extract the information required for pedestrian detection from text features, we divide the processing of text features into two branches. The first branch is the image-guided cross attention, which uses the infrared feature F i and visible light characteristics F v As the query vector Q in the attention, the text feature is used as the key vector K and the value vector V. The infrared feature and the visible light feature actively extract the key semantic information in the text, such as "enhance the edge" or "improve the contrast". The second branch is the text-guided cross attention, which uses the text feature T as the query vector and the infrared feature F as the value vector. i and visible light characteristics F v As the key vector K and value vector V, the text features are used to locate the blurred areas in the image and strengthen the task-related local features.

[0202] In the first branch, we will F i ,F v As the vector to be queried (Q), T as the key vector (K) and value vector (V), perform cross attention calculation:

[0203]

[0204] in, Infrared signature indicating that text information is injected into the first branch; Indicates the visible light feature of the text information injected into the first branch; d1 indicates The dimension size of Query information represented as infrared features; is the key value of T to infrared features; It is expressed as the value of T for infrared characteristics; query information represented as visible features; is the key value of T for visible light features; It is expressed as the value of T for visible light features; W represents a learnable matrix; the Softmax function is an activation function that can normalize a numerical vector into a probability distribution vector.

[0205] In the second branch, we use T as the query vector (Q), F i ,F v As the key vector (K) and value vector (V), perform cross attention calculation:

[0206]

[0207] Among them, F i 2 Infrared signature indicating that text information is injected into the second branch; Indicates the visible light feature of the text information injected into the second branch; d2 indicates The dimension size of Query information represented as infrared features; is the key value of T to infrared features; It is expressed as the value of T for infrared characteristics; query information represented as visible features; is the key value of T for visible light features; Represented as the value of T for visible light features; W represents a learnable matrix. In order to fully aggregate text information, we will and The concatenation is performed on the channel dimension and the result is input into three feature extraction layers to obtain the fine-tuned infrared features. and fine-tuned visible light characteristics Using only branch one text suggestion cannot adapt to the image spatial structure, and using only branch two image details cannot fully absorb the text semantics.

[0208] We will fine-tune the infrared features and fine-tuned visible light characteristics Splicing is performed on the channel dimension (the splicing operation is as follows: Where C(·) represents the splicing operation on the channel dimension) and is sent to the fusion feature decoder to reconstruct the fine-tuned fusion image I′ f .

[0209] In order to make the fine-tuned fusion image I′ fTo meet the needs of pedestrian safety risk identification tasks, we introduced the existing pedestrian safety risk identification network, whose link is https: / / github.com / ultralytics / yolov5 (its network architecture is YOLOv5, with its backbone network, neck network and prediction head frozen) to supervise the fusion image. We will I′ f Input into the pedestrian safety risk identification network to obtain the pedestrian safety risk identification prediction results In order to make the pedestrian safety risk identification results Can be as close as possible to its true label y gt Closely, we introduce the loss l used in training the pedestrian safety risk recognition network pd Constrain the text-driven feature reconciliation module. pd The same loss function is used in the YOLOv5 training process. The YOLOv5 loss function consists of three parts: positioning loss, confidence loss, and classification loss. The total loss formula is:

[0210] l pd =λ coord ·L CIoU +λ obj ·L obj +λ cls ·L cls

[0211] Among them, the positioning loss weight λ coord =0.05, target confidence loss weight λ obj =1.0, classification loss weight λ cls =0.5. Positioning loss L CIoU , confidence loss L obj , classification loss L cls As shown below:

[0212]

[0213] Among them, IoU is the intersection-over-union ratio of the predicted box and the real box; ρ(b,b st ) is the center point b of the predicted box and the center point b of the real box st The Euclidean distance between the two boxes; c is the diagonal length of the minimum enclosed area of ​​the two boxes; Used to measure the aspect ratio difference, v is the aspect ratio of the predicted box and the real box.

[0214]

[0215] Among them, S 2 is the number of grids in the feature map; B is the number of bounding boxes predicted for each grid; is the true label (0 or 1), indicating whether the j-th bounding box of the i-th grid contains the target; C ij is the prediction confidence, which represents the model's probability prediction value that the j-th bounding box of the i-th grid contains the target.

[0216]

[0217] Among them, S 2 is the number of grids in the feature map, and C is the number of categories; is the true category probability; P ic is the predicted category probability.

[0218] Application Examples

[0219] This example uses the following three public datasets for experiments: LLVIP: contains paired infrared and visible light images, suitable for low-light scenes; 3 FD: Multi-scene and multi-modal dataset, supporting target detection tasks in complex environments; MSRS: A general dataset for infrared and visible light image fusion. The training set is 2000 pairs of infrared and visible light images randomly selected from the LLVIP dataset, and data enhancement is performed by random flipping, rotation, and data augmentation; the fusion performance test set is 2000 pairs of infrared and visible light images randomly selected from the LLVIP dataset, and data augmentation is performed by random flipping, rotation, and data augmentation. 3 200 pairs of images are randomly selected from FD and MSRS, and the pedestrian detection performance test set only uses the LLVIP dataset.

[0220] To verify the effectiveness of the method of the present invention, the method proposed in the present invention is divided into two training stages. In the first training stage, paired infrared images and visible light images are input, and a fused image is output. The goal is to optimize the fusion quality through intensity loss and edge loss. In the second training stage, the basic fused image generated by the frozen fusion network and the problem prompt of the multimodal large language model (LLaVA) are input, and the fused image optimized by the text-driven feature harmonization module (i.e., the fine-tuned fused image) is output. It is supervised by the pre-trained pedestrian detection network to ensure that the fused image improves the performance of downstream tasks.

[0221] In the first and second training phases, we train the fusion network and text-driven feature blending module on the public dataset LLVIP, respectively. Specifically, we randomly select 2,000 pairs of infrared and visible light images from the LLVIP dataset as the training set. To enhance the diversity of training samples, we randomly flip and rotate the training dataset. In addition, we select 2,000 pairs of infrared and visible light images from the LLVIP dataset, and randomly rotate the training dataset. 3 200 pairs of infrared and visible light images were randomly selected from FD and MSRS as test sets to verify the fusion performance and pedestrian safety risk recognition performance of the proposed method. 3FD and MSRS were used to verify the fusion performance, and LLVIP was used to verify the pedestrian safety risk identification performance. In the first training phase, we trained the fusion network. In the second training phase, we froze the parameters of the fusion network and trained the text-driven feature blending module. In both training phases, the Adam optimizer was used to update the network parameters, with a batch size of 16 and a learning rate of 1×10. -3 The total number of training rounds in the first and second training phases was 100 and 200, respectively. Furthermore, the hyperparameter was set to 0.2. The proposed method was implemented using the Pytorch framework and trained on a single NVIDIA RTX A6000 GPU.

[0222] This paper uses five commonly used objective evaluation indicators to quantitatively analyze the fusion performance of the method proposed in the patent, namely edge preservation (Q AB / F ), Chen-Varshney index (Q CV ), structural similarity (Q SSIM ), average gradient index (Q AG ) and the sum of the correlation differences (Q SCD ).

[0223] Q AB / F It is used to measure the degree to which the edge information in the source image is retained in the fused image. The larger the value, the less loss of texture detail information in the fused image.

[0224]

[0225] in, Q BF (i, j) is the same; Γ A , Γ B is the Sobel edge intensity map of source images A and B; Γ F is the Sobel edge intensity map of the fused image F;

[0226] w A (i,j)=(Γ A (i,j)) α , w B (i,j)=(Γ B (i,j)) α , α is the weight parameter (usually 1); K is a stability constant used to prevent the denominator from being zero, usually K = 1; W, H are the width and height of the image.

[0227] Q CV From the perspective of human visual perception, the quality of the fused image is evaluated. The smaller the value, the more consistent the fused image is with human visual perception.

[0228]

[0229] Among them, Q contrast is the local contrast similarity, which is calculated as the local contrast difference between the source image and the fused image. where σ A ,σ F Where is the standard deviation in the local window, is a constant; Q structure is the local structural similarity, which is calculated as the gradient direction consistency between the source image and the fused image. in represents the gradient operator; λ is the balancing weight (usually 0.5); C represents a constant.

[0230] Q SSIM The brightness, contrast and structural similarity between the fused image and the source image are measured. The larger the value, the smaller the information difference between the fused image and the source image.

[0231]

[0232] Among them, μ A 、μ F is the mean brightness of image A and image F, obtained by calculating the average value of the pixel values ​​in the local window; σ A , σ F is the standard deviation, σ AF is the covariance; C1=(k1L) 2 、C2=(k2L) 2 , L is the pixel value range (such as 255), k1=0.01, k2=0.03.

[0233] Q AG It is used to measure the richness of gradient information in the fused image. A larger value means richer gradient information in the fused image.

[0234]

[0235] in, are the horizontal and vertical gradients of the fused image; W and H are the image width and height respectively.

[0236] Q SCD The degree of information loss during the fusion process is measured by calculating the difference map between the source image and the fused image. A larger value indicates less distortion in the fused image.

[0237]

[0238] Among them, ρ(A c,F) is the Pearson correlation coefficient between the c-th channel of the source image A and the fused image F, which is calculated as Where cov(X,Y) is the covariance, σ X ,σ Y is the standard deviation, ρ(B c ,F) Similarly, the correlation coefficient ρ∈[-1,1], the closer the value is to 1, the stronger the positive correlation is; C is the number of image channels (such as RGB image C=3).

[0239] Q AB / F , Q SSIM , Q AG and Q SCD It is a positive evaluation index, that is, the larger the value of the evaluation index is, the better the fusion performance of the comparison method is; Q CV It is an inverse evaluation index, that is, the smaller the value of the evaluation index, the better the fusion performance of the comparison method. In addition, in order to objectively evaluate the performance of the fusion results obtained by the method proposed in this paper on the pedestrian safety risk identification task, we adopted the mean average precision (mAP) with an intersection-over-union ratio threshold of 0.5 to 0.95, which is widely used in the field of pedestrian safety risk identification. 50→95 ) for quantitative analysis.

[0240] The proposed method is compared quantitatively and qualitatively with AUIF, DATFuse, IVFWSR, LRRNet, MLFusion and TIMFusion on the LLVIP and M3FD datasets to verify the advantages of the proposed method in fusion performance. Figure 4 It can be seen from the enlarged area in that the method proposed in the present invention not only retains the texture details in the visible light image, but also highlights the thermal radiation information in the infrared image. Compared with the existing SOTA method, the fused image obtained by the method proposed in the present invention has clearer local details, and the brightness and contrast at the global level are also higher. This not only brings higher image quality, but also enables better identification of targets in downstream tasks. This can also be verified in the results of quantitative evaluation, as shown in Table 1 and Table 2. It can be seen from the quantitative results that the method proposed in the present invention has a better performance in Q CV The value on is low, Q AB / F and Q AG The values ​​of are ranked first, which means that the edge information in the fused image is richer and more consistent with human visual perception.

[0241] Table 1 Comparison of objective evaluation of different methods on the LLVIP dataset; ↑ indicates that the indicator is a positive indicator, and the larger the value, the better the performance; ↓ indicates that the indicator is a negative indicator, and the smaller the value, the better the performance.

[0242] method <![CDATA[Q AB / F ↑]]> <![CDATA[Q CV ↓]]> <![CDATA[Q SSIM ↑]]> <![CDATA[Q AG ↑]]> <![CDATA[Q SCD ↑]]> AUIF 0.3869 610.74 1.2016 3.5256 1.3413 DATFuse 0.4548 453.42 1.3130 3.1243 1.3351 IVFWSR 0.2925 512.77 1.2348 2.5252 1.1235 LRRNet 0.4426 534.89 1.3022 2.4625 0.9999 MLFusion 0.3239 523.41 1.2624 2.1613 0.9966 TIMFusion 0.2325 845.75 1.1742 2.1761 0.5368 Method of the present invention 0.5845 287.43 1.3441 3.9867 1.5462

[0243] Table 2 Objective evaluation comparison of different methods on the M3FD dataset

[0244] method <![CDATA[Q AB / F ↑]]> <![CDATA[Q CV ↓]]> <![CDATA[Q SSIM ↑]]> <![CDATA[Q AG ↑]]> <![CDATA[Q SCD ↑]]> AUIF 0.5425 852.56 1.3003 6.6735 1.5353 DATFuse 0.4854 563.57 1.3067 4.8326 1.3461 IVFWSR 0.4532 722.22 1.2735 3.5628 1.2452 LRRNet 0.5164 579.55 1.3113 4.5624 1.3461 MLFusion 0.4253 689.44 1.2835 4.4527 1.2687 TIMFusion 0.5352 616.16 1.2872 4.3336 1.2004 Method of the present invention 0.5856 454.45 1.3095 6.4561 1.6187

[0245] In order to make the results of the fusion network perform better in downstream tasks, a common practice is to freeze the parameters of the fusion network and retrain the downstream task network based on the fusion results generated by it. This type of method is called the "retraining method". In order to verify the performance of the method proposed in this paper in the pedestrian safety risk identification task, we compared it with the "retraining method" quantitatively and qualitatively. Figure 4 It can be seen that in the pedestrian safety risk identification results of other methods, there are cases where the pedestrian's body is not completely contained in the detection frame, and even some detection frames contain a large amount of background areas unrelated to the pedestrian, showing insufficient detection accuracy. In contrast, in the pedestrian safety risk identification result diagram of the method proposed in the present invention, the irrelevant areas in the detection frame are significantly reduced, and the detection frame position is more accurate. This advantage is also fully reflected in the quantitative evaluation results, as shown in Table 3. mAP of the method proposed in the present invention 50→95 Ranked first, it outperforms the comparison method in the pedestrian safety risk identification task. In summary, the proposed method, without requiring retraining of the pedestrian safety risk identification network, demonstrates superior performance compared to methods that require network retraining, fully demonstrating its effectiveness and advantages in pedestrian safety risk identification tasks.

[0246] Table 3 Quantitative comparison results of the pedestrian safety risk identification performance with the “retraining method”.

[0247] method <![CDATA[mAP 50→95 ↑]]> AUIF 74.4 DATFuse 74.3 IVFWSR 72.9 LRRNet 73.8 MLFusion 73.6 TIMFusion 74.0 Method of the present invention 75.0

[0248] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A pedestrian risk identification method based on infrared and visible light image fusion driven by a large model, characterized by: The steps include: Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing; Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder; Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2; Step 4, text feature generation: The preliminary fused image generated after the first stage of training and the question prompt are input into the large language and vision assistant to obtain text features; Step 5: Build a text-driven feature reconciliation module; The text-driven feature reconciliation module consists of a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and leaky ReLU activation functions, and the three feature extraction layers are connected sequentially; The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network; Step 6, second stage training: Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module; The pedestrian safety risk identification network is the YOLOv5 model; Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

2. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 1 is characterized in that: The steps include: Step 1: Image acquisition and preprocessing: Acquire a single-channel infrared image of the power scene and an RGB visible light image of the same power scene, and perform preprocessing; Step 2, building a fusion network: the fusion network includes an infrared image feature encoder, a visible light image feature encoder and a fusion feature decoder; The infrared image feature encoder and the visible light image feature encoder are both connected to the fusion feature decoder; The infrared image feature encoder is used to extract features from the preprocessed infrared image to obtain infrared features; The visible light image feature encoder is used to extract features from the preprocessed visible light image to obtain visible light features; The infrared features and visible light features are spliced ​​in the channel dimension, and the spliced ​​results are input into the fusion feature decoder for fusion to obtain the preliminary fusion image I f ; Step 3, first stage training: perform the first stage training on the fusion network constructed in Step 2; Step 4, text feature generation: the initial fusion image I generated after the first stage of training f Input it together with the question prompt into a large language and visual assistant to obtain text features T; Step 5: Build a text-driven feature reconciliation module; The text-driven feature reconciliation module includes a dual-branch cross attention and three feature extraction layers; each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions, and the three feature extraction layers are connected sequentially; The dual-branch cross attention includes the first-branch cross attention and the second-branch cross attention; the output of the first-branch cross attention and the output of the second-branch cross attention are both connected to the input of the first feature extraction layer among the three feature extraction layers; The first branch is image-guided cross attention, which is used to enable infrared features and visible light features to actively extract semantic information from the text, obtaining infrared features injected with text information in the first branch and visible light features injected with text information in the second branch; The second branch is a text-guided cross-attention method, which is used to locate the blurred areas in the infrared and visible light feature images using text features and strengthen local features, thereby obtaining infrared features with text information injected into the second branch and visible light features with text information injected into the second branch. Then, the infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain fine-tuned infrared features and fine-tuned visible light features; The text-driven feature reconciliation module is also connected to the fusion feature decoder in the fusion network; Step 6, second stage training: Freeze the first-order pre-trained fusion network parameters, train the text-driven feature reconciliation module, input the fine-tuned fusion image into the pedestrian detection network to obtain the prediction result image, and perform iterative training on the prediction image and its corresponding true label loss to obtain the trained text-driven feature reconciliation module; The pedestrian safety risk identification network is a YOLOv5 model; Step 7, pedestrian risk identification: Obtain a single-channel infrared image of the power scene to be detected and an RGB visible light image of the same power scene. After preprocessing according to the same method as Step 1, a fine-tuned fusion image is obtained through a fusion network, a large language and vision assistant, and a text-driven feature blending module. The fusion image is then input into the pedestrian detection network for pedestrian risk identification to obtain the pedestrian risk identification result.

3. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 2 is characterized in that: In Step 1, the specific method of preprocessing is: The single-channel infrared image and the RGB three-channel visible light image are randomly cropped to a size of 256×256, and then normalized and then data augmented.

4. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 3 is characterized in that: The specific method of normalization operation is: first, subtract the minimum pixel value in the image from each pixel; Then divide it by the minimum pixel value in the image; finally, perform an overall offset, that is, all pixel values ​​× 2, and then subtract 1, so that the image pixel range becomes between -1 and 1; Data augmentation methods include random flipping and random rotation.

5. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 2 is characterized in that: In Step 2, the infrared image feature encoder, the visible light image feature encoder, and the fusion feature decoder all include three feature extraction layers connected sequentially; Each feature extraction layer is composed of a stack of convolutional layers, batch normalization layers, and Leaky ReLU activation functions; The Leaky ReLU activation function in the last feature extraction layer of the fusion feature decoder is replaced by the Tanh activation function; The preprocessed infrared image I i Input the infrared image feature encoder to extract features and obtain infrared features F i ; The preprocessed visible light image I v Input the visible light image feature encoder to extract features and obtain the visible light feature F v ; F i and F v Perform splicing operation on the channel dimension and input it into the fusion feature decoder to reconstruct the preliminary fusion image I f ; The splicing operation is as follows: C(F i ,F v )=F f , where C(·) represents the concatenation operation on the channel dimension, C is the channel dimension, H is the image height, W is the image width, F f The result of splicing.

6. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 5 is characterized in that: The convolution kernel size of the convolution layer is 3×3 and the stride is 1.

7. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 5 is characterized in that: The specific method of Step 3 is: Step 3.1: Use AdamW optimizer to optimize the parameters in the fusion network; Step 3.2: When training the fusion network, the loss function uses the fusion loss When the preset total number of training rounds is reached, the training of the fusion network is stopped.

8. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 7 is characterized in that: Fusion loss The calculation method is as follows: Among them, ε represents the hyperparameter used to balance each sub-loss item; Strength loss for: Edge loss for: Among them, H and W represent the height and width of the preliminary fusion image respectively, ||·||1 represents the l1 norm, Represents the Sobel edge extraction operator.

9. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 2 is characterized in that: In Step 5: The first branch is the image-guided cross attention, which takes the infrared feature F i and visible light characteristics F v As the query vector Q in the attention, the text feature is used as the key vector K and the value vector V, so that the infrared feature and the visible light feature can actively extract the key semantic information in the text, and the infrared feature injected with the text information in the first branch and the visible light feature injected with the text information in the first branch are obtained; The second branch is the text-guided cross attention, which takes the text feature T as the query vector and the infrared feature F i and visible light characteristics F v As the key vector K and value vector V, the text feature is used to locate the blurred area in the image, and the local features are strengthened to obtain the infrared features injected with text information in the second branch and the visible light features injected with text information in the second branch; The infrared features injected with text information in the first branch, the visible light features injected with text information in the first branch, the infrared features injected with text information in the second branch, and the visible light features injected with text information in the second branch are spliced ​​in the channel dimension and input into three feature extraction layers for extraction to obtain the fine-tuned infrared features. Fine-tuned visible light characteristics The text-driven feature reconciliation module obtains and Perform splicing operation on the channel dimension and send it to the fusion feature decoder to obtain the fine-tuned fusion image I′ f .

10. The pedestrian risk identification method based on large model driven infrared and visible light image fusion according to claim 2 is characterized in that: The specific method of Step 6 is: Step 6.

1. Use the AdamW optimizer to train and optimize the parameters in the text-driven feature reconciliation module. Step 6.2, the fine-tuned fusion image I′ f Input into the pedestrian safety risk identification network to obtain the pedestrian safety risk identification result; Among them, the loss function of the text-driven feature reconciliation module is the same as the loss function of the pedestrian risk recognition network. pd The same process is repeated until the training reaches 200 rounds.

Images