A cross-modal pedestrian detection method based on intermediate state images

Abstract

The application discloses a cross-modal pedestrian detection method based on an intermediate state image, and comprises the following steps: S1, pre-processing an input image; S2, inputting the image pre-processed in the step S1 into an intermediate state generation network to generate an intermediate state image between a visible light modality and a thermal infrared modality; and S3, inputting the intermediate state image generated in the step S2 and the visible light image into a double-flow fusion network to perform feature extraction, feature fusion and pedestrian detection, so as to obtain the position of the pedestrian in the image. The method generates the intermediate state image by using the intermediate state image generation network, the intermediate state image combines the advantages of the visible light modality and the thermal infrared modality, can better assist the pedestrian detection of the visible light, and does not need to use the thermal infrared image during the test, and is more in line with the demand of actual application. The accuracy of the method is higher than that of the prior art, meanwhile, the feature map fusion and strengthening on the feature maps of different sizes can improve the performance of the pedestrian detection on different sizes.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and more specifically to a cross-modal pedestrian detection method based on intermediate-state images. Background Technology

[0002] Pedestrian detection utilizes computer vision technology to determine the presence and precise location of pedestrians in video or image sequences. It is a fundamental core technology for computer vision tasks such as pedestrian re-identification, pedestrian tracking, and pedestrian counting, and a pioneering technology for applications such as intelligent robots, autonomous driving, and surveillance. Unlike other objects, pedestrians exhibit highly unpredictable movement. During filming, occlusion between people and objects, or between people themselves, can easily lead to the loss or even severe obscuring of information about the pedestrian's body. Furthermore, changes in scale and background during pedestrian movement further complicate pedestrian detection. Therefore, pedestrian detection is a challenging yet valuable and popular research topic.

[0003] Most pedestrian detection methods rely solely on visible light surveillance data. However, visible light surveillance only provides clear, easily distinguishable images under bright daylight conditions. In low-visibility scenarios such as nighttime, rain, or fog, the image quality deteriorates significantly, as pedestrians blend into the surrounding environment, making accurate detection difficult. This is where thermal infrared (TII) images from thermal infrared sensors come in. In low-visibility conditions, TII images, based on the difference in thermal radiation between pedestrians and their environment, provide clear pedestrian outlines, effectively distinguishing pedestrian areas from the background. In high-visibility conditions, visible light images provide auxiliary detection information such as color and texture. Due to the inherent complementarity between these two modalities, cross-modal pedestrian detection, which combines both modalities, has become a key focus for researchers.

[0004] In recent years, cross-modal pedestrian detection has been extensively studied. The mainstream approach utilizes the complementarity between two modalities (visible light and thermal infrared). This fusion method is generally based on a two-stream network structure, with visible light and thermal infrared images as inputs to the two branches respectively, followed by fusion detection. This two-stream structure means that both training and testing require paired visible light and thermal infrared images as input. However, in practical applications, most road surveillance is currently visible light-based. Installing paired thermal infrared cameras or replacing them with integrated visible light-thermal infrared cameras is costly and fails to provide the necessary visible light-thermal infrared image pairs for two-stream networks. Therefore, a method is urgently needed that utilizes existing thermal infrared-visible light image datasets only during training, while requiring only visible light images for testing in real-world scenarios. This would compensate for the shortcomings of visible light images with thermal infrared images, meet practical application needs, and significantly reduce costs. Summary of the Invention

[0005] To overcome the shortcomings of the above technologies, this invention provides a cross-modal pedestrian detection method based on intermediate state images, which can be widely applied to intelligent security monitoring, autonomous driving and other scenarios.

[0006] This invention first utilizes an intermediate state generation network to generate an intermediate state image that combines the advantages of visible light and thermal infrared modes for visible light images. Then, the intermediate state image and the visible light image are input into a two-stream fusion network for feature extraction and pedestrian detection.

[0007] Terminology Explanation:

[0008] 1. BCE Loss: Binary Cross Entropy Loss.

[0009] 2. GIOU Loss: Generalized Intersection over Union Loss.

[0010] 3. CFT: Cross-modal Fusion Transformer.

[0011] The technical solution adopted by this invention to overcome its technical problems is:

[0012] A cross-modal pedestrian detection method based on intermediate state images includes the following steps:

[0013] S1. Preprocess the input image;

[0014] S2. Input the image after preprocessing in step S1 into the intermediate state generation network to generate an intermediate state image between the visible light mode and the thermal infrared mode;

[0015] S3. Input the intermediate image and visible light image generated in step S2 into the dual-stream fusion network for feature extraction, feature fusion and pedestrian detection to obtain the location of pedestrians in the image.

[0016] Furthermore, step S1 specifically includes:

[0017] S1.1 Crop the input image to a preset size;

[0018] S1.2 Transform the image in the HSV color space, where H represents the color of the image, S represents the intensity of the color, and V represents the brightness of the image.

[0019] S1.3. Perform data augmentation on the image processed in step S1.2 by random horizontal flipping.

[0020] Furthermore, in step S1, the input image is divided into training mode and testing mode:

[0021] If it is in training mode, the input images are pairs of visible light-thermal infrared images;

[0022] If it is in test mode, the input image is a single visible light image.

[0023] Furthermore, step S2 specifically includes:

[0024] S2.1. Encode the input image after preprocessing in step S1 using an encoder to obtain the main features of the input image;

[0025] S2.2 Transform the main features of the obtained input image into intermediate image features that combine thermal infrared image and visible light image;

[0026] S2.3 Decode the features of the transformed intermediate image to obtain the intermediate image.

[0027] Furthermore, in step S2, the intermediate state generation network is constrained by three types of loss functions: generation loss, discrimination loss, and recurrence loss.

[0028] (1) Generation loss:

[0029] In the cross-modal pedestrian detection task, since the input is a pair of visible light-thermal infrared images, the generator is guided to generate an intermediate image that combines the features of the real visible light image and thermal infrared image by means of the generation loss constraint. The generation loss is as follows (1):

[0030]

[0031] In equation (1), the generation loss consists of two parts, namely and in, This represents the generator loss when the input is a visible light image. The generator loss is represented by the input thermal infrared image; E represents the BCE loss; i V Represents a visible light image, i F Represents a thermal infrared image; p(i V p(i) represents the distribution of visible light image data. F ) represents the distribution of thermal infrared image data; i V ~p(i V () indicates that a visible light image is randomly selected from the visible light images in the dataset, i F ~p(i F This indicates that a thermal infrared image is randomly selected from the thermal infrared images in the dataset. This represents the BCE loss when a visible light image is randomly selected from the visible light images in the dataset. G represents the BCE loss of randomly selecting a thermal infrared image from the dataset; V G represents a generator whose input is a visible light image. F This indicates a generator whose input is a thermal infrared image; G V (i V ) indicates that the input visible light image is fed into the generator G. V To generate intermediate images, G F (i F ) indicates that the input thermal infrared image is fed into the generator G. F To generate an intermediate image;

[0032] (2) Determine the loss:

[0033] The discriminant loss is used to constrain and guide two discriminators, enabling them to distinguish between real images and images generated by the generator. The two discriminators include a discriminator for distinguishing thermal infrared images and a discriminator for distinguishing visible light images. The discriminant loss is as follows (2):

[0034]

[0035] Among them, D F L(G) represents the discriminator used to distinguish thermal infrared images. V D F ,I V ,I F Loss-guided constraint D FThis enables it to accurately distinguish between real thermal infrared images and thermal infrared images generated by the generator based on the input visible light image; D V It is a discriminator used to distinguish visible light images, L(G) F D V ,I F ,I V Loss-guided constraint D V This enables it to accurately distinguish between a real visible light image and a visible light image generated by the generator based on an input thermal infrared image; D F (i F This indicates that the thermal infrared image is input into the discriminator D. F D V (i V This indicates that the visible light image is input into the discriminator D. V ;D F (G V (i V The process is represented as: G V (i V Input a visible light image into generator G V An intermediate image is generated, and then the generated intermediate image is input into the discriminator D. F Through discriminator D F The output results are used to determine whether the generated intermediate image contains information from a thermal infrared image; D V (G F (i F The process is represented as: G F (i F Input thermal infrared image to generator G F An intermediate image is generated, and then the generated intermediate image is input into the discriminator D. V Through discriminator D V The output is used to determine whether the generated intermediate image contains information from a visible light image.

[0036] (3) Cyclic loss:

[0037] The intermediate state generation network adopts a recurrent generation network structure, using two generators: one for generating a visible light image and the other for generating a generator G. V and the generator G whose input is a thermal infrared image F Generator G V Input a visible light image and generate a thermal infrared image; Generator G F Input a thermal infrared image to generate a visible light image; when the original visible light image is processed by the generator G... F Generate intermediate images using discriminator D FDetermine whether the generated intermediate image retains the effective features of the thermal infrared image: If the generated intermediate image retains the features of both the thermal infrared image and the visible light image, then input the generated intermediate image into the generator G. F Based on the visible light features preserved in the intermediate image, the generator G... F Restored to a visible light image; when the original thermal infrared image is processed by generator G V Generate intermediate images using discriminator D V Determine whether the generated intermediate image retains the effective features of the visible light image: If the generated intermediate image retains the features of both the visible light image and the thermal infrared image, then input the generated intermediate image back into the generator G. V Based on the thermal infrared image features retained on the intermediate state image, the generator G... V Restored to a thermal infrared image; cyclic loss is used for the constraint generator G. V and generator G F The quality is as follows (3):

[0038]

[0039] Cyclic losses include two parts: "visible light → intermediate state → visible light" and "thermal infrared → intermediate state → thermal infrared". G F (G V (i V )) represents the input intermediate image to the generator G F The intermediate image is restored to a visible light image to determine whether it contains visible light information. V (G F (i F )) represents the input intermediate image to the generator G V The intermediate image is restored to a thermal infrared image to determine whether it contains thermal infrared information.

[0040] Furthermore, in step S2, the total loss function of the intermediate state generation network includes the weighted sum of the generator's generation loss, the discriminator's discrimination loss, and the recurrence loss, as shown in equation (4):

[0041] minL total =λ gen L gen +λ dis L dis +λ cyc L cyc (4)

[0042] Where, λ gen , λ dis , λ cycThese are used to weigh the proportions of generation loss, discrimination loss, and cyclical loss, respectively.

[0043] Furthermore, step S3 specifically includes:

[0044] S3.1 Input the intermediate image and the visible light image into the dual-stream fusion network, and extract the features of the intermediate image and the visible light image respectively;

[0045] S3.2. Simultaneously input the features of the intermediate image and the visible light image into the CFT module, and perform the following operations:

[0046] First, the two feature maps are jointly operated on at the channel dimension;

[0047] Then, by utilizing the self-attention mechanism in the CFT module, the effective pedestrian feature parts in the visible light image and intermediate modal image are enhanced to form an optimized fused feature map;

[0048] Finally, the fused feature map is added to the feature maps of the intermediate image and the visible light image, respectively, to obtain the enhanced and corrected intermediate image features and the enhanced and corrected visible light image features;

[0049] S3.3. The enhanced and corrected intermediate image features and the enhanced and corrected visible light image features are fused together, and then pedestrian detection is performed to locate the pedestrian's position.

[0050] Furthermore, in step S3, three types of loss functions are used during the training of the dual-stream fusion network: localization loss, classification loss, and confidence loss, as detailed below:

[0051] (1) Positioning loss:

[0052] The localization loss is used to constrain the detector so that it can accurately detect the size of the pedestrian and generate a rectangular box that exactly surrounds the pedestrian target. The localization loss is defined as follows (5):

[0053]

[0054] In the above formula, s 2 This represents the number of grid cells in the feature map generated during the process of predicting the bounding box of a pedestrian; N represents the number of predicted boxes generated in each grid cell. This indicates whether the target is present at the j-th detection box in the i-th grid of the feature map. If a pedestrian target is present, The value is assigned as 1, otherwise it is 0, meaning that only the localization loss of positive samples is calculated; This represents the GIOU loss of the i-th grid; GIOU i represents the GIOU value of the i-th grid; B represents the name of the bounding box. This represents the true bounding box of the i-th grid. This represents the prediction box for the i-th grid. Let represent the minimum bounding rectangle of the ground truth box and the predicted box of the i-th grid; where a rectangle represents a rectangular box, a bounding box represents a rectangle containing a pedestrian, a detection box represents a rectangle detected by the detector, a predicted box represents a rectangle containing a pedestrian after filtering the detection boxes, and a ground truth box represents a real rectangle containing a pedestrian.

[0055] (2) Classification loss:

[0056] The classification loss is defined as follows (6):

[0057]

[0058] Classification loss L cls The calculation is performed using cross-entropy, p i (c) represents the probability that the true sample belongs to class c, where c represents a specific class among all classifications; p i (c) represents the probability that the true sample belongs to class c. If the sample does not belong to class c, then p i (c) = 0; This represents the probability that the network predicts a sample to be of class c.

[0059] In pedestrian detection, if the dataset only has one category, pedestrian, then the classification loss is not calculated, and L... cls The value is set to 0; if the dataset is divided into at least several categories such as walking people, stationary people, and cyclists, then the classification loss needs to be calculated.

[0060] (3) Confidence loss:

[0061] The confidence score is used to calculate the loss for all samples, including negative samples, as shown in equation (7):

[0062]

[0063] In the above formula, L noobj L represents the confidence loss for negative samples. obj This represents the confidence loss for positive samples; and... on the contrary, This indicates whether a target is present at the j-th detection box of the i-th grid on the feature map. If no pedestrian target is found, the value is 1; otherwise, it is 0. i This indicates the true confidence level of the prediction box. This indicates the confidence level of the predicted bounding box as determined by the network.

[0064] Furthermore, the total loss function of the dual-stream fusion network is the weighted sum of the localization loss, classification loss, and confidence loss, as shown in equation (8):

[0065] minL total =λ box L box +λ cls L cls +λ conf L conf (8)

[0066] Where, λ box , λ cls , λ conf These are used to weigh the proportions of localization loss, classification loss, and confidence loss, respectively.

[0067] The beneficial effects of this invention are:

[0068] This invention uses existing thermal infrared-visible dual-spectral data only during the training phase, and does not use thermal infrared data during testing. The method includes: encoding the input image using an encoder to obtain the main features of the input image; transforming the obtained main features of the input image into intermediate image features that meet the requirements for pedestrian detection and combine infrared and visible light images; and decoding the transformed features to obtain the intermediate image. Then, the intermediate image and the visible light image are input into a dual-stream fusion network, and features of the intermediate image and the visible light image are extracted from the two branches of the network, respectively. Feature fusion and detection are performed on feature maps at three different scales (specifically 160×160×128, 80×80×256, and 40×40×512). This method utilizes an intermediate image generation network to generate intermediate images, which combine the advantages of visible light and thermal infrared modes, better assisting in visible light pedestrian detection. Furthermore, it does not require the use of thermal infrared images during testing, making it more suitable for practical applications. By comparing the results with existing pedestrian detection methods that only use visible light, the accuracy of the method of this invention is higher than that of existing methods. At the same time, the method of this invention fuses and enhances feature maps at different scales, which can improve the performance of pedestrian detection at different sizes. Attached Figure Description

[0069] Figure 1 This is a block diagram illustrating the principle of the cross-modal pedestrian detection method based on intermediate-state images as described in an embodiment of the present invention.

[0070] Figure 2 This is a schematic diagram illustrating the principle of generating intermediate images according to an embodiment of the present invention.

[0071] Figure 3 This is a schematic diagram of the intermediate state generation network described in an embodiment of the present invention.

[0072] Figure 4 This is a schematic diagram of the dual-stream fusion network structure described in an embodiment of the present invention.

[0073] Figure 5 This is a comparison chart regarding pedestrian missed detection as described in an embodiment of the present invention.

[0074] Figure 6 This is a comparison chart of pedestrian confidence levels as described in an embodiment of the present invention.

[0075] Figure 7 This is a comparison diagram regarding pedestrian false detection as described in an embodiment of the present invention. Detailed Implementation

[0076] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following are merely exemplary and do not limit the scope of protection of the present invention.

[0077] This invention discloses a cross-modal pedestrian detection method based on intermediate state images, such as... Figure 1 As shown, it includes the following steps:

[0078] S1. Preprocess the input image.

[0079] Step S1 specifically includes:

[0080] S1.1. Crop the input image to the preset size.

[0081] S1.2. Transform the image in the HSV color space, where H represents the image color, S represents the intensity of the color, and V represents the image brightness. This step randomly transforms the image's color, intensity, and brightness, increasing image diversity and preventing model overfitting.

[0082] S1.3. Perform data augmentation on the image processed in step S1.2 by random horizontal flipping. The purpose of this data augmentation step is to enhance the diversity of the image, make it more realistic, and prevent the model from overfitting.

[0083] In step S1, the input image is divided into training mode and test mode: if it is training mode, the input image is a pair of visible light-thermal infrared images; if it is test mode, the input image is a single visible light image.

[0084] S2. The image preprocessed in step S1 is input into the intermediate state generation network to generate an intermediate state image between the visible light mode and the thermal infrared mode.

[0085] like Figure 2 As shown, step S2 specifically includes:

[0086] S2.1. The input image after preprocessing in step S1 is encoded by an encoder to obtain the main features of the input image.

[0087] S2.2 Transform the main features of the obtained input image into intermediate image features that combine thermal infrared and visible light images. The intermediate image features are more in line with the needs of pedestrian detection.

[0088] S2.3 Decode the features of the transformed intermediate image to obtain the intermediate image.

[0089] Furthermore, in step S2, the intermediate state generation network is constrained by three types of loss functions: generation loss, discrimination loss, and recurrence loss.

[0090] (1) Generation loss:

[0091] Unlike general image generation tasks, in cross-modal pedestrian detection tasks, since the input is a pair of visible light-thermal infrared images, the generator is guided to generate a high-quality intermediate image that combines features of real visible light and thermal infrared images by means of a generation loss constraint. The generation loss is as follows (1):

[0092]

[0093] In equation (1), the generation loss consists of two parts, namely and in, This represents the generator loss when the input is a visible light image. The generator loss is represented by the input thermal infrared image; E represents the BCE loss; i V Represents a visible light image, i F Represents a thermal infrared image; p(i V p(i) represents the distribution of visible light image data. F ) represents the distribution of thermal infrared image data; i V ~p(i V () indicates that a visible light image is randomly selected from the visible light images in the dataset, i F ~p(i F This indicates that a thermal infrared image is randomly selected from the thermal infrared images in the dataset. This represents the BCE loss when a visible light image is randomly selected from the visible light images in the dataset. G represents the BCE loss of randomly selecting a thermal infrared image from the dataset; V G represents a generator whose input is a visible light image. FThis indicates a generator whose input is a thermal infrared image; G V (i V ) indicates that the input visible light image is fed into the generator G. V To generate intermediate images, G F (i F ) indicates that the input thermal infrared image is sent to the generator G. F To generate an intermediate image.

[0094] (2) Determine the loss:

[0095] The purpose of the discriminant loss is to constrain and guide the two discriminators to distinguish between real images and images generated by the generator. The two discriminators include a discriminator for distinguishing thermal infrared images and a discriminator for distinguishing visible light images. The discriminant loss is as follows (2):

[0096]

[0097] Among them, D F L(G) represents the discriminator used to distinguish thermal infrared images. F D F ,I V ,I F Loss-guided constraint D F This enables it to accurately distinguish between real thermal infrared images and thermal infrared images generated by the generator based on the input visible light image; D V It is a discriminator used to distinguish visible light images, L(G) F D V ,I F ,I V Loss-guided constraint D V This enables it to accurately distinguish between a real visible light image and a visible light image generated by the generator based on an input thermal infrared image; D F (i F This indicates that the thermal infrared image is input into the discriminator D. F D V (i V This indicates that the visible light image is input into the discriminator D. V ;D F (G V (i V The process is represented as: G V (i V Input a visible light image into generator G V An intermediate image is generated, and then the generated intermediate image is input into the discriminator D. F Through discriminator D F The output results are used to determine whether the generated intermediate image contains information from a thermal infrared image; D V (GF (i F The process is represented as: G F (i F Input thermal infrared image to generator G F An intermediate image is generated, and then the generated intermediate image is input into the discriminator D. V Through discriminator D V The output is used to determine whether the generated intermediate image contains information from a visible light image.

[0098] (3) Cyclic loss:

[0099] In this embodiment, to obtain the best transformation method, the intermediate state generation network adopts a cyclic generation network structure, such as... Figure 3 As shown, x and y represent the visible light image and the thermal infrared image, respectively. The two generators used are generator G, which takes the visible light image as input. V and the generator G whose input is a thermal infrared image F Generator G V Input a visible light image, generate a thermal infrared image; generator G F Input a thermal infrared image, generate a visible light image. D F and D V These represent the discriminator used to distinguish thermal infrared images and the discriminator used to distinguish visible light images, respectively. and This is an intermediate image. and These are the restored visible light image and thermal infrared image, respectively. When the original visible light image x is processed by generator G... F Generate intermediate image Using discriminator D F Determine the generated intermediate image Does it retain the effective features of the thermal infrared image: if the generated intermediate image The intermediate image retains the characteristics of both thermal infrared and visible light images. Input to generator G F Based on the visible light features preserved in the intermediate image, the generator G... F Restored to visible light image When the original thermal infrared image y is processed by generator G V Generate intermediate image Using discriminator D V Determine the generated intermediate image Does it retain the effective features of the visible light image: if the generated intermediate image The intermediate image retains the characteristics of both visible light and thermal infrared images. Input to generator G V Based on the thermal infrared image features retained on the intermediate state image, the generator G... V Restored to thermal infrared image Compared to a typical generative network, a cyclic loss is introduced to constrain the generator G. V and generator G F The quality is as follows (3):

[0100]

[0101] Cyclic losses include two parts: "visible light → intermediate state → visible light" and "thermal infrared → intermediate state → thermal infrared". G F (G V (i V )) represents the input intermediate image to the generator G F The intermediate image is restored to a visible light image to determine whether it contains visible light information. V (G F (i F )) represents the input intermediate image to the generator G V The intermediate image is restored to a thermal infrared image to determine whether it contains thermal infrared information.

[0102] The total loss function of the intermediate state generation network, as described above, includes the weighted sum of the generator's generation loss, the discriminator's discrimination loss, and the recurrence loss, as shown in equation (4):

[0103] minL total =λ gen L gen +λ dis L dis +λ cyc L cyc (4)

[0104] Where, λ gen , λ dis , λ cyc These are used to weigh the generation loss, discrimination loss, and recurrent loss, respectively. Optimizing the recurrent generation network can be solved using existing network optimization algorithms (such as gradient descent). Therefore, by optimizing the total loss function of the intermediate generation network and updating the generation and discrimination networks, we can effectively optimize the generator performance while improving the discriminator's discrimination ability. As the detector's discrimination ability improves, the images generated by the generator must be more similar to the original image to fool the discriminator.

[0105] S3. Input the intermediate image and visible light image generated in step S2 into the dual-stream fusion network for feature extraction, feature fusion and pedestrian detection to obtain the location of pedestrians in the image.

[0106] Step S3 specifically includes:

[0107] S3.1. Input the intermediate image and visible light image generated in step S2 into the dual-stream fusion network to extract features from the intermediate image and visible light image respectively. That is, the intermediate image and visible light image are fed together into the dual-stream feature extraction network to assist the visible light image in pedestrian detection.

[0108] S3.2. Simultaneously input the features of the intermediate image and the visible light image into the CFT module, and perform the following operations:

[0109] First, the two feature maps are jointly operated on in the channel dimension. Specifically, the features of the intermediate image and the features of the visible light image are connected in the same dimension. For example, two three-channel feature maps are jointly operated on in the channel dimension to obtain a six-channel feature map.

[0110] Then, by utilizing the self-attention mechanism in the CFT module, the effective pedestrian feature parts in the visible light image and intermediate modal image are enhanced to form an optimized fused feature map;

[0111] Finally, the fused feature map is added to the feature maps of the intermediate image and the visible light image, respectively, to obtain the enhanced and corrected features of the intermediate image and the enhanced and corrected features of the visible light image.

[0112] S3.3. The enhanced and corrected intermediate image features and the enhanced and corrected visible light image features are fused together, and then pedestrian detection is performed to locate the pedestrian's position.

[0113] like Figure 4As shown, M represents the obtained feature map, V and F represent visible light and thermal infrared respectively, and P is the result fed into the detection after fusion. This embodiment uses the CFT (Cross-Modal Fusion Transformer) module three times. Under the guidance of the CFT module, the dual-stream fusion network can extract global context information for each modality during the feature extraction stage. More importantly, the self-attention mechanism in the CFT module is utilized to discover more effective feature parts in the visible light image and intermediate state image, and fuse them to form an optimized fused feature map. The fused feature map is then added to the feature maps of the two branches respectively, thereby strengthening and correcting the features of each modality. Simultaneously, the three-fold fusion of multispectral features can refine the single-spectral features. Since the fused features are usually more discriminative than the spectral features, the enhanced spectral features should also be more discriminative than the original spectral features.

[0114] Furthermore, in step S3, three types of loss functions are used during the training of the dual-stream fusion network: localization loss, classification loss, and confidence loss, as detailed below:

[0115] (1) Positioning loss:

[0116] Localization loss is the most important loss in cross-modal pedestrian detection, involving the localization of pedestrians. Localization loss is used to constrain the detector so that it accurately detects the size of pedestrians and generates a bounding box that perfectly surrounds the pedestrian target. The definition of localization loss is as follows (5):

[0117]

[0118] In the above formula, s 2 This represents the number of grid cells in the feature map generated during the process of predicting the bounding box of a pedestrian (e.g., 13*13, 26*26, 52*52); N represents the number of predicted boxes generated in each grid cell. This indicates whether the target is present at the j-th detection box in the i-th grid of the feature map. If a pedestrian target is present, The value is assigned as 1, otherwise it is 0, meaning that only the localization loss of positive samples is calculated; This represents the GIOU loss of the i-th grid; GIOU i represents the GIOU value of the i-th grid; B represents the name of the bounding box. This represents the true bounding box of the i-th grid. This represents the prediction box for the i-th grid. Let represent the minimum bounding rectangle of the ground truth bounding box and the predicted bounding box of the i-th grid; where a rectangle represents a rectangular box, a bounding box represents a rectangle containing a pedestrian, a detection box represents a rectangle detected by the detector, a predicted box represents a rectangle containing a pedestrian after filtering the detection boxes, and a ground truth bounding box represents a real rectangle containing a pedestrian.

[0119] (2) Classification loss:

[0120] same, This indicates whether the target is present at the j-th detection box in the i-th grid of the feature map. If a pedestrian target is present, The value is assigned as 1 if the value is positive and 0 otherwise, meaning that only the localization loss of positive samples is calculated. The classification loss is defined as follows (6):

[0121]

[0122] Classification loss L cls The calculation is performed using cross-entropy, p i (c) represents the probability that the true sample belongs to class c, where c represents a specific class among all classifications; p i (c) represents the probability that the true sample belongs to class c. If the sample does not belong to class c, then p i (c) = 0; This represents the probability that the network predicts a sample to be of class c.

[0123] In pedestrian detection, if the dataset only has one category, pedestrian, then the classification loss is not calculated, and L... cls The value is set to 0; if the dataset is divided into at least several categories such as walking people, stationary people, and cyclists, then the classification loss needs to be calculated.

[0124] (3) Confidence loss:

[0125] Unlike localization loss and classification loss, confidence loss is calculated for all samples, including negative samples, as shown in equation (7):

[0126]

[0127] In the above formula, L noobj L represents the confidence loss for negative samples. obj This represents the confidence loss for positive samples; and... on the contrary, This indicates whether a target is present at the j-th detection box of the i-th grid on the feature map. If no pedestrian target is found, the value is 1; otherwise, it is 0. i This indicates the true confidence level of the prediction box. This indicates the confidence level of the predicted bounding box as determined by the network.

[0128] Furthermore, the total loss function of the dual-stream fusion network is the weighted sum of the localization loss, classification loss, and confidence loss, as shown in equation (8):

[0129] minL total =λ box L box +λ cls L cls +λ conf L conf (8)

[0130] Where, λ box , λ cls , λ conf These are used to weigh the localization loss, classification loss, and confidence loss, respectively. Optimizing the two-stream fusion network can be solved using existing network optimization algorithms (such as gradient descent). Therefore, by optimizing the total loss function of the two-stream fusion network and updating the pedestrian detection network, we can effectively reduce the differences between the two modalities, improve the detector's detection performance, and obtain better pedestrian detection results.

[0131] The method of this invention is compared with the YOLOv5 method on the FLIR dataset, as shown in Table 1. Using mAP50 and mAP as evaluation metrics, it can be seen that the method of this invention outperforms YOLOv5 in both mAP50 and mAP. The method of this invention converges after 50 epochs of training. For fairness in the comparison, YOLOv5 was trained for 100 epochs until convergence, yielding mAP50 and mAP. As shown in the table, the method of this invention still outperforms the already converged YOLOv5 in both evaluation metrics.

[0132] Table 1 compares the method of this invention with the YOLOv5 method on the FLIR dataset.

[0133] method Training image type Test Image Type Number of training cycles mAP50 (%) mAP(%) Yolov5l Visible light images Visible light images 50 20.5 8.22 Yolov5l Visible light images Visible light images 100 38.8 12.6 Method of the present invention Visible light image + thermal infrared image Visible light images 50 41.4 13.8

[0134] Below, we further demonstrate the effectiveness of the method of this invention by showcasing the visualization results of the model detection. Figures 5-7 The effectiveness of the method of the present invention is illustrated by comparing three sets of graphs: pedestrian missed detection, pedestrian confidence score, and pedestrian false detection. Figures 5-7 In the diagram, the left column shows the detection results obtained by training and testing with visible light images using existing methods; the right column shows the detection results obtained by training with both thermal infrared and visible light images using the method of this invention, and then testing with visible light images alone. The boxes in both the left and right columns represent predicted pedestrian bounding boxes, with the predicted category being "person," and the number after the category representing the confidence level, ranging from [0.0, 1.0].

[0135] from Figure 5As can be seen, specifically... Figure 5 (a) Figure 5 (b) Figure 5 In (c), the predicted bounding boxes in the left column all missed some pedestrians, while the predicted bounding boxes in the right column using the method of this invention can detect these missed pedestrians and assign them a higher confidence level. For example, Figure 5 In (a), the pedestrian under the streetlight was not detected by the visible light detector, but the detector of the method of the present invention successfully detected the pedestrian and gave a pedestrian confidence score of 0.57.

[0136] Figure 6 In a daytime scene with normal lighting, Figure 6 In (a), the confidence level of the visible light detector in the left column for the middle pedestrian instance is 0.39, while the confidence level of the detector of the present invention in the right column is 0.79 (0.39 in the left column and 0.79 in the right column). For Figure 6 For the cyclist in (b), it can be seen that the confidence level given by the visible light detector in the left column is 0.38, while the confidence level given by the detector of the present invention in the right column is 0.53 (0.38 in the left column and 0.53 in the right column). Figure 6 (c) shows a pedestrian on the right. It can be seen that the confidence level given by the visible light detector in the left column is 0.64, while the confidence level given by the detector of the method of the present invention in the right column is 0.74 (0.64 in the left column and 0.74 in the right column).

[0137] Figure 7 In, such as Figure 7 (a) In the left column, the visible light detector incorrectly detected the streetlight in the middle and the bushes next to the car as pedestrian instances; as shown... Figure 7 (b) Left column: The visible light detector incorrectly identifies a car trunk as a pedestrian instance; as shown... Figure 7 (c) In the left column, the visible light detector incorrectly detects the palm tree as a pedestrian instance. However, in the detector output of the method of the present invention (right column), the street lamp in the middle, the grass next to the car, the car trunk, and the palm tree are not identified as pedestrian instances.

[0138] In summary, the method of the present invention can converge faster, locate pedestrians in images more accurately, and has higher accuracy.

[0139] The above description only outlines the basic principles and preferred embodiments of the present invention. Those skilled in the art can make many changes and modifications based on the above description, and these changes and modifications should fall within the protection scope of the present invention.

Claims

1. A method for cross-modality pedestrian detection based on intermediate state images, characterized in that, The method comprises the following steps: S1, preprocessing the input image; S2, inputting the image preprocessed in step S1 into an intermediate state generation network to generate an intermediate state image between the visible light modality and the thermal infrared modality, wherein a total loss function of the intermediate state generation network comprises a weighted sum of a generation loss of a generator, a discrimination loss of a discriminator and a cycle loss; S3, inputting the intermediate state image generated in step S2 and the visible light image into a double-flow fusion network to perform feature extraction, feature fusion and pedestrian detection, and obtaining the position of the pedestrian in the image, wherein a total loss function of the double-flow fusion network is a weighted sum of a positioning loss, a classification loss and a confidence loss; Step S3 specifically comprises: S3.1, inputting the intermediate state image and the visible light image into the double-flow fusion network to extract the features of the intermediate state image and the visible light image respectively; S3.2, inputting the features of the intermediate state image and the visible light image into a CFT module at the same time, and performing the following operations: First, the two feature maps are jointly operated in the channel dimension; Then, the CFT module is used to strengthen the effective pedestrian feature part in the visible light image and the intermediate modality image by using the self-attention mechanism in the CFT module, and an optimized fusion feature map is formed; Finally, the fusion feature map is added to the feature map of the intermediate state image and the feature map of the visible light image respectively, so as to obtain the strengthened and corrected intermediate state image feature and the strengthened and corrected visible light image feature; S3.3, fusing the strengthened and corrected intermediate state image feature and the strengthened and corrected visible light image feature, and then performing pedestrian detection to locate the position of the pedestrian.

2. The method of claim 1, wherein the method further comprises: Step S1 specifically comprises: S1.1, cropping the input image into a preset size; S1.2, transforming the image in the HSV color space, wherein H represents the color of the image, S represents the depth of the color of the image, and V represents the brightness of the image; S1.3, performing data enhancement on the image processed in step S1.2 by using random horizontal flipping.

3. The method of claim 1, wherein the method further comprises: In step S1, the input image is divided into a training mode and a test mode: If it is the training mode, the input image is a pair of visible light-thermal infrared image pairs; If it is the test mode, the input image is a single visible light image.

4. The method of claim 1, wherein the method further comprises: Step S2 specifically comprises: S2.1, encoding the input image preprocessed in step S1 by using an encoder to obtain the main features of the input image; S2.2, transforming the obtained main features of the input image into intermediate state image features combined with the thermal infrared image and the visible light image; S2.3, decoding the transformed intermediate state image features to obtain the intermediate state image.

5. The method of claim 1, wherein, In step S2, the intermediate state generation network is constrained by three types of loss functions, and the three types of loss are: generation loss, discrimination loss and cycle loss; (1) Generation loss: In the cross-modal pedestrian detection task, since the input is a pair of visible light-thermal infrared image pairs, the generation loss constraint is used to guide the generator to generate an intermediate state image combined with the features of the real visible light image and the thermal infrared image, and the generation loss is as follows: （1） In formula (1), the generation loss consists of two parts, i.e. and wherein, represents the generator loss with input of visible light image, represents the generator loss with input of thermal infrared image; represents the BCE loss; represents the visible light image, represents the thermal infrared image; represents the distribution of visible light image data, represents the distribution of thermal infrared image data; represents a visible light image randomly taken from the visible light images in the data set, represents a thermal infrared image randomly taken from the thermal infrared images in the data set; represents the BCE loss of a visible light image randomly taken from the visible light images in the data set, represents the BCE loss of a thermal infrared image randomly taken from the thermal infrared images in the data set; represents the generator with input of visible light image, represents the generator with input of thermal infrared image; represents inputting a visible light image to the generator to generate an intermediate state image, represents inputting a thermal infrared image to the generator to generate an intermediate state image; (2) Discrimination loss: The discriminant loss is used to constrain the two discriminators to distinguish the real picture and the picture generated by the generator, wherein the two discriminators include a discriminator for distinguishing the thermal infrared image and a discriminator for distinguishing the visible light image, and the discriminant loss is as follows formula (2): （2） wherein, represents a discriminator for distinguishing thermal infrared images, loss-guided constraint so as to accurately distinguish between real thermal infrared images and thermal infrared images generated by the generator from input visible light images; is a discriminator for distinguishing visible light images, loss-guided constraint so as to accurately distinguish between real visible light images and visible light images generated by the generator from input thermal infrared images; represents the process of inputting a thermal infrared image to a discriminator , represents the process of inputting a visible light image to a discriminator ; represents the process as follows: inputting a visible light image to a generator to generate an intermediate state image, and then inputting the generated intermediate state image to a discriminator , and determining whether the generated intermediate state image contains information of a thermal infrared image through the output result of the discriminator ; represents the process as follows: inputting a thermal infrared image to a generator to generate an intermediate state image, and then inputting the generated intermediate state image to a discriminator , and determining whether the generated intermediate state image contains information of a visible light image through the output result of the discriminator ; (3) Cycle loss: The intermediate state generation network adopts a cyclic generation network structure, and two generators are adopted , one is a generator inputting a visible light image , and the other is a generator inputting a thermal infrared image ; the generator inputs a visible light image to generate a thermal infrared image; the generator inputs an original thermal infrared image to generate an intermediate state image, and a discriminator is used to determine whether the generated intermediate state image retains effective features of the thermal infrared image; if the generated intermediate state image retains both the features of the thermal infrared image and the features of the visible light image, the generated intermediate state image is input to the generator again, and the visible light image is restored from the retained visible light features on the intermediate state image through the generator ; the generator inputs an original visible light image to generate an intermediate state image, and a discriminator is used to determine whether the generated intermediate state image retains effective features of the visible light image; if the generated intermediate state image retains both the features of the visible light image and the features of the thermal infrared image, the generated intermediate state image is input to the generator again, and the thermal infrared image is restored from the retained thermal infrared image features on the intermediate state image through the generator ; and a cycle loss is used to constrain the generator and the generator , as shown in the following formula (3): （3） The cycle loss includes cycle loss of two parts of "visible light→ intermediate state→ visible light" and "thermal infrared→ intermediate state→ thermal infrared", representing inputting the intermediate state image to the generator , restoring the intermediate state image to the visible light image to determine whether it contains visible light information, representing inputting the intermediate state image to the generator , restoring the intermediate state image to the thermal infrared image to determine whether it contains thermal infrared information.

6. The method of claim 5, wherein the method further comprises: In step S2, the total loss function of the intermediate state generation network includes the weighted sum of the generation loss of the generator, the discriminant loss of the discriminator and the cycle loss, as follows formula (4): （4） wherein, respectively used for weighing the proportion of the generation loss, the discrimination loss, and the cycle loss.

7. The method of claim 1, wherein the method further comprises: In step S3, three types of loss functions are used in the training process of the double-flow fusion network, including positioning loss, classification loss and confidence loss, as follows: (1) Positioning loss: The positioning loss is used to constrain the detector to accurately detect the size of the pedestrian, and generate a rectangular frame that exactly surrounds the pedestrian target, and the definition of the positioning loss is as follows formula (5): （5） In the above formula, represents the number of grids in the feature map generated in the process of predicting the bounding box of the pedestrian; represents the number of predicted boxes generated in each grid; represents whether the target is contained in the first grid at the first detection box on the feature map, if the pedestrian target is contained, the value is 1, otherwise 0, that is, only the positioning loss of the positive sample is calculated; represents the GIOU loss of the first grid; represents the GIOU value of the first grid; represents the alias of the bounding box, represents the real box of the first grid; represents the predicted box of the first grid; represents the minimum circumscribed rectangle of the real box and the predicted box of the first grid; wherein, the rectangular frame represents the square frame of the rectangle, the bounding box represents the rectangular frame containing the pedestrian, the detection box represents the rectangular frame detected by the detector, the predicted box represents the rectangular frame including the pedestrian obtained after screening the detection box, and the real box represents the rectangular frame actually containing the pedestrian. (2) Classification loss: The definition of the classification loss is as follows formula (6): (6) classification loss is computed in the form of cross-entropy, (c) represents the probability that the true sample is of a class, represents a specific class among all classes; (c) represents the probability that the true sample is of a class, if the sample does not belong to this class, ; (c) represents the probability that the network predicts the sample to be of a class; In pedestrian detection, if the dataset only has one category, pedestrian, then the classification loss is not calculated. The value is set to 0; if the dataset is divided into at least several categories such as walking people, stationary people, and cyclists, then the classification loss needs to be calculated. (3) Confidence loss: The confidence loss calculates the loss of all samples, including negative samples, as follows formula (7): （7） In the above formula, This represents the confidence loss for negative samples. This represents the confidence loss for positive samples; and... on the contrary, This indicates that if the first [characteristic] on the feature map... The first grid If a target is detected at a given detection box, the value is set to 1 if there is no pedestrian target, and 0 otherwise. This indicates the true confidence level of the prediction box. This indicates the confidence level of the predicted bounding box as determined by the network.

8. The method of claim 1, wherein, The total loss function of the double-flow fusion network is the weighted sum of the positioning loss, the classification loss and the confidence loss, as follows formula (8): （8） wherein, respectively used to weigh the proportion of positioning loss, classification loss, and confidence loss.

Images