Model construction and training method and device, and target detection method, device and system

Abstract

The invention relates to a model construction and training method and device, and a target detection method, device and system. The model training method comprises the steps that a first network model is constructed, and the first network model is used for carrying out multi-level feature fusion on a visible light image and an infrared image of a target to obtain a fusion feature map; a second network model is constructed, and the second network model is used for obtaining the fusion feature map to carry out multi-scale feature fusion; and a third network model is constructed, the third network model is used for executing bounding box regression and classification and performing target detection, the target detection model comprises a first network model, a second network model and a third network model, and the trained target detection model is used for realizing target detection. According to the invention, for the problem of feature fusion opportunity of different modes, a multi-level feature fusion scheme is adopted to enhance feature interaction of the visible light image and the infrared image, so that fusion among different modes is more thorough.

Description

Technical Field

[0001] This disclosure relates to the field of target detection technology, and in particular to a model building and training method and apparatus, a target detection method, apparatus and system. Background Technology

[0002] Human-centered computer vision tasks (such as pedestrian detection, pedestrian re-identification, and pedestrian retrieval) have attracted widespread attention in the past. Among these tasks, pedestrian detection is one of the most fundamental and has broad application prospects. Pedestrian detection is a computer vision technique that locates and identifies pedestrians based on their features. Besides its direct applications in video surveillance and autonomous driving, it is also a prerequisite for other vision tasks. For example, pedestrian re-identification and pedestrian retrieval first require the identification of all existing pedestrians.

[0003] Pedestrian detection technology has made significant progress, generally focusing on two issues: pedestrian location and size, and pedestrian classification results. Common pedestrian detection methods can be categorized into three types: (a) methods based on traditional feature extraction to generate pedestrian candidate regions and deep classification features, such as Deformable Part Model (DPM), decision forests, and gradient histograms; (b) deep learning-based methods employing end-to-end training, including joint learning of pedestrian candidate box generation and class classification; and (c) hybrid methods combining the advantages of traditional weak classifiers and deep learning classifiers. These typically use traditional methods to detect some candidate boxes and then use a deep learning classifier to further refine the pedestrian candidate boxes.

[0004] Most pedestrian detection methods in related technologies focus on visible light cameras (i.e., RGB images, or color images). They typically employ the three primary colors (red, green, and blue) imaging principle, meaning three different filters are used within the camera, each corresponding to a different wavelength of light, and images are captured for each of the three channels. Finally, image processing algorithms combine the three channel images to reconstruct the original visible light image. Visible light cameras perform well during the day but perform poorly at night. Summary of the Invention

[0005] The inventors discovered through research that related technologies consider incorporating infrared cameras to improve the performance of pedestrian detection algorithms at night. However, there are issues with the timing of feature fusion between infrared and visible light modalities in these technologies: most multispectral fusion methods perform fusion at the pixel, feature, and decision levels, but different fusion timings have different advantages and disadvantages. Pixel-level fusion incorporates multimodal information in the preprocessing step, requiring high spatiotemporal alignment of the initial data, while feature-level fusion is used in the feature processing layer. Due to the large number of layers in deep convolutional neural networks, it is difficult to determine at which feature layer to fuse the data. Decision-level fusion focuses on fusing the results of a two-stream network at the decision level, which results in the loss of a large amount of intermediate semantic features.

[0006] In view of at least one of the above technical problems, this disclosure provides a model construction and training method and apparatus, a target detection method, apparatus and system. For the problem of feature fusion timing of different modalities, a multi-level feature fusion scheme is adopted to enhance the feature interaction between visible light images and infrared images, so as to make the fusion between different modalities more thorough.

[0007] According to one aspect of this disclosure, a method for constructing an object detection model is provided, comprising:

[0008] A first network model is constructed, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and multi-level feature fusion is performed on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0009] A second network model is constructed, wherein the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps;

[0010] A third network model is constructed, wherein the third network model is used to perform bounding box regression and classification for object detection. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to achieve object detection.

[0011] In some embodiments of this disclosure, the first network model includes multiple stage sub-models connected in series, each stage sub-model including:

[0012] The visible light extraction sub-model is used to extract features from the visible light image of the target or the feature map of the visible light image output by the previous stage sub-model, so as to obtain the feature map of the visible light image of the current stage sub-model.

[0013] The infrared extraction sub-model is used to extract features from the infrared image of the target or the feature map of the infrared image of the previous stage sub-model to obtain the feature map of the infrared image of the current stage sub-model.

[0014] The fusion sub-model is used to fuse the feature maps of the visible light image and the infrared image of the current stage sub-model.

[0015] In some embodiments of this disclosure, constructing the first network model includes:

[0016] Multiple stage sub-models are constructed in series. The first network model includes n stage sub-models, where n is a natural number greater than 1. Each stage sub-model includes a visible light extraction sub-model and an infrared extraction sub-model. The visible light extraction sub-model and infrared extraction sub-model of the first stage sub-model are used to perform convolution operations and downsampling on the input visible light image and infrared image to generate the feature map of the first stage. The visible light extraction sub-model and infrared extraction sub-model of the i-th stage sub-model are used to perform convolution operations and downsampling on the feature map generated by the (i-1)-th stage sub-model to generate the feature map of the i-th stage, where i is a natural number greater than 1 and less than or equal to n.

[0017] In some embodiments of this disclosure, constructing the cascaded multiple-stage sub-model includes:

[0018] Multiple modality alignment modules are constructed, wherein the fusion sub-model includes a modality alignment module, which is connected to the visible light extraction sub-model and the infrared extraction sub-model respectively. The modality alignment module is used to perform feature space alignment on the feature maps of the visible light image and the infrared image.

[0019] In some embodiments of this disclosure, the modal alignment module includes:

[0020] The feature concatenation model is used to concatenate feature maps of visible light images and infrared images to generate concatenated feature maps.

[0021] A first transformation network is used to process the cascaded feature map to obtain a first scaling factor and a first offset, wherein the first scaling factor and the first offset are the scaling factor and offset in the feature space transformation relationship from infrared image to visible light image;

[0022] The second transformation network is used to process the cascaded feature map to obtain a second scaling factor and a second offset, wherein the second scaling factor and the second offset are the scaling factor and offset in the feature space transformation relationship from visible light image to infrared image.

[0023] In some embodiments of this disclosure, the modal alignment module further includes:

[0024] The feature space adaptation operation model is used to perform element-wise multiplication of the feature map of the infrared image with the first scaling factor, and then sum the feature map of the infrared image with the first offset to obtain the feature map of the infrared image after adaptive transformation; and to perform element-wise multiplication of the feature map of the visible light image with the second scaling factor, and then sum the feature map of the visible light image with the second offset to obtain the feature map of the visible light image after adaptive transformation.

[0025] The convolution model is used to sum the feature maps of the visible light image and the adaptively transformed infrared image element-wise, and then perform a convolution operation to obtain the transfer feature map of the visible light image. Similarly, the feature maps of the infrared image and the adaptively transformed visible light image are summed element-wise and then convolved to obtain the transfer feature map of the infrared image.

[0026] In some embodiments of this disclosure, constructing the cascaded multiple-stage sub-model includes:

[0027] Multiple differential information fusion modules are constructed, wherein the fusion sub-model includes a differential information fusion module, which is connected to the visible light extraction sub-model and the infrared extraction sub-model respectively. The differential information fusion module is used to enhance the relationship between the feature maps of the visible light image and the infrared image by utilizing the differences between the feature maps of the visible light image and the infrared image.

[0028] In some embodiments of this disclosure, the differential information fusion module includes:

[0029] A differential feature determination model is used to determine the first differential feature based on the difference between the feature map of the infrared image and the feature map of the visible light image, wherein the feature map of the infrared image is the output of the infrared extraction sub-model of the previous stage sub-model, and the feature map of the visible light image is the output of the visible light extraction sub-model of the previous stage sub-model; and to determine the second differential feature based on the difference between the feature maps of the visible light image and the infrared image.

[0030] An auxiliary feature determination model is used to calculate a feature map to assist the visible light image based on the first difference feature, and to calculate a feature map to assist the infrared image based on the second difference feature.

[0031] The differential fusion model is used to sum the element-wise features of the feature map of the visible light image and the feature map used to assist the visible light image to determine the fused feature map of the visible light image, and to sum the element-wise features of the feature map of the infrared image and the feature map used to assist the infrared image to determine the fused feature map of the infrared image.

[0032] In some embodiments of this disclosure, the auxiliary feature determination model includes:

[0033] The weight factor generation module is used to perform global average pooling on the first difference feature, scale and restore the channels through convolution, and generate the first weight factor through an activation function; and to perform global average pooling on the second difference feature, scale and restore the channels through convolution, and generate the second weight factor through an activation function.

[0034] The auxiliary feature determination module is used to perform element-level feature integration between the first weight factor and the feature map of the infrared image to obtain a feature map used to assist the visible light image; and to perform element-level feature integration between the second weight factor and the feature map of the visible light image to obtain a feature map used to assist the infrared image.

[0035] In some embodiments of this disclosure, constructing the cascaded multiple-stage sub-model includes:

[0036] A stitching module is constructed, wherein the fusion sub-model of the nth stage sub-model includes the stitching module, and the stitching module is used to stitch together the feature map of the visible light image and the feature map of the infrared image generated by the (n-1)th stage sub-model.

[0037] In some embodiments of this disclosure, the fusion sub-model includes a modality alignment module and a differential information fusion module, and the construction of the cascaded multi-stage sub-model includes:

[0038] An input determination module is constructed, wherein the input determination module is used to determine that the input of the visible light extraction sub-model in the i-th stage sub-model is the sum of the original input of the visible light extraction sub-model, the transfer feature map of the visible light image after passing through the modal alignment module of the (i-1)-th stage sub-model, and the fusion feature map of the visible light image after passing through the differential information fusion module of the (i-1)-th stage sub-model; and to determine that the input of the infrared extraction sub-model in the i-th stage sub-model is the sum of the original input of the infrared extraction sub-model, the transfer feature map of the infrared image after passing through the modal alignment module of the (i-1)-th stage sub-model, and the fusion feature map of the infrared image after passing through the differential information fusion module of the (i-1)-th stage sub-model, wherein the first network model includes n stage sub-models, where n is a natural number greater than 1, and i is a natural number greater than 1 and less than or equal to n.

[0039] In some embodiments of this disclosure, the object detection model construction method further includes:

[0040] A fourth network model is constructed, wherein the fourth network model is connected to the first network model, and is used to determine whether the current image was taken during the day or at night by using the feature map output by the visible light extraction sub-model of the first network model as input, and to adjust the feature map output by each stage sub-model in the first network model by using internal feature representation.

[0041] In some embodiments of this disclosure, constructing the fourth network model includes:

[0042] A feature representation generation module is constructed, wherein the feature representation generation module is used to perform average pooling on the feature map of the visible light image of the visible light extraction sub-model in the first stage sub-model, and then input it into two fully connected layers, and generate internal feature representation through the first activation function;

[0043] An adjustment parameter generation module is constructed, wherein the adjustment parameter generation module is used to generate the first adjustment parameter and the second adjustment parameter of each stage sub-model based on the internal feature representation;

[0044] A feature map adjustment module is constructed, wherein the feature map adjustment module is used to adjust the feature map of each stage sub-model according to the first adjustment parameter and the second adjustment parameter of each stage sub-model.

[0045] In some embodiments of this disclosure, the adjustment parameter generation module includes:

[0046] The first adjustment parameter generation submodule is used to determine the weights and biases corresponding to the first adjustment parameters of the k-th stage sub-model; and to determine the first adjustment parameters based on the weights and biases corresponding to the first adjustment parameters, the internal feature representation, and the first activation function.

[0047] The second adjustment parameter generation submodule is used to determine the weights and biases corresponding to the second adjustment parameters of the k-th stage sub-model; and to determine the second adjustment parameters based on the weights and biases corresponding to the second adjustment parameters, the internal feature representation, and the first activation function.

[0048] In some embodiments of this disclosure, the feature map adjustment module includes:

[0049] The third adjustment parameter generation submodule is used to determine the third adjustment parameter based on the difference between 1 and the first adjustment parameter;

[0050] The feature map adjustment submodule is used to multiply the third adjustment parameter with the feature map of the k-th stage sub-model, then sum it with the second adjustment parameter, and input it into the first activation function to obtain the adjusted feature map of the k-th stage sub-model.

[0051] In some embodiments of this disclosure, constructing the second network model includes:

[0052] A top-down structure is constructed, wherein the top-down structure is used to upsample the feature map output by the first network model, and the upsampled feature map is merged with the feature maps output by the visible light extraction sub-model and the infrared extraction sub-model in the first network model after feature extraction;

[0053] A bottom-up structure is constructed, wherein the bottom-up structure is used to stitch together multi-layer feature maps, and the multi-layer feature maps are feature maps that have been processed by the top-down structure at different times.

[0054] In some embodiments of this disclosure, constructing the third network model includes:

[0055] Multiple detection heads of different sizes are constructed to cover targets of different scales. Each detection head is used to predict whether a target exists within the grid and to determine target information, including the target's location, size, and category.

[0056] According to another aspect of this disclosure, a method for training an object detection model is provided, comprising:

[0057] The target detection model is trained using image training data, so that the trained target detection model can be used to perform target detection. The target detection model includes a first network model, a second network model, and a third network model.

[0058] In some embodiments of this disclosure, training the target detection model using image data includes:

[0059] The first network model is subjected to feature extraction and fusion training. The first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image. Multi-level feature fusion is performed on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0060] The second network model is trained to perform multi-scale feature fusion, wherein the second network model is used to obtain multi-scale feature maps of the fused feature map and to perform multi-scale feature fusion on the multi-scale feature maps;

[0061] The third network model is trained for object detection. The third network model is used to perform bounding box regression and classification to perform object detection. The object detection model includes the first network model, the second network model and the third network model. The trained object detection model is used to achieve object detection.

[0062] In some embodiments of this disclosure, the target detection model is constructed using the target detection model construction method described in any of the above embodiments.

[0063] According to another aspect of this disclosure, a target detection method is provided, the target detection method being performed by a target detection model, wherein the target detection method includes:

[0064] Receive the visible light and infrared images at the current moment;

[0065] Feature extraction is performed on the visible light image and infrared image at the current moment to obtain the feature map of the visible light image and the feature map of the infrared image respectively;

[0066] Multi-level feature fusion is performed on the feature maps of visible light images and infrared images to obtain a fused feature map.

[0067] Object detection is performed by performing bounding box regression and classification on the fused feature maps.

[0068] In some embodiments of this disclosure, the multi-level feature fusion of the feature maps of the visible light image and the infrared image includes:

[0069] Feature space alignment is performed on the feature maps of visible light images and infrared images.

[0070] In some embodiments of this disclosure, the feature space alignment of the feature maps of the visible light image and the infrared image includes:

[0071] The feature maps of visible light images and infrared images are concatenated to generate a concatenated feature map.

[0072] The cascaded feature map is processed to obtain a first scaling factor and a first offset, wherein the first scaling factor and the first offset are the scaling factor and offset in the feature space transformation relationship from infrared image to visible light image;

[0073] The cascaded feature map is processed to obtain a second scaling factor and a second offset, wherein the second scaling factor and the second offset are the scaling factor and offset in the feature space transformation relationship from visible light image to infrared image.

[0074] In some embodiments of this disclosure, the feature space alignment of the feature maps of the visible light image and the infrared image further includes:

[0075] After performing element-wise multiplication of the feature map of the infrared image with the first scaling factor, and then summing it element-wise with the first offset, the feature map of the infrared image after adaptive transformation is obtained.

[0076] After performing element-wise multiplication of the feature map of the visible light image with the second scaling factor, and then summing it element-wise with the second offset, the feature map of the visible light image after adaptive transformation is obtained.

[0077] The feature map of the visible light image and the feature map of the adaptively transformed infrared image are summed element-wise and then convolved to obtain the transfer feature map of the visible light image.

[0078] The feature map of the infrared image is obtained by summing the feature map of the adaptively transformed visible light image element-wise and then performing a convolution operation.

[0079] In some embodiments of this disclosure, the multi-level feature fusion of the feature maps of the visible light image and the infrared image includes:

[0080] The relationship between the feature maps of visible light images and infrared images can be enhanced by utilizing the differences between their feature maps.

[0081] In some embodiments of this disclosure, enhancing the relationship between the feature maps of a visible light image and an infrared image by utilizing the differences between their feature maps includes:

[0082] The first differential feature is determined based on the difference between the feature maps of the infrared image and the visible light image, where the feature map of the infrared image is the output of the infrared extraction sub-model of the previous stage sub-model, and the feature map of the visible light image is the output of the visible light extraction sub-model of the previous stage sub-model; the second differential feature is determined based on the difference between the feature maps of the visible light image and the infrared image.

[0083] The feature map used to assist the visible light image is calculated based on the first difference feature, and the feature map used to assist the infrared image is calculated based on the second difference feature.

[0084] The visible light image feature map is determined by summing the element-level features of the feature map used to assist the visible light image; the infrared image feature map is determined by summing the element-level features of the feature map used to assist the infrared image.

[0085] In some embodiments of this disclosure, the target detection model includes a first network model, which comprises multiple stage sub-models connected in series. Each stage sub-model is used to extract features from the visible light image and infrared image of the target, respectively, to obtain feature maps of the visible light image and the infrared image; the target detection method further includes:

[0086] Using the infrared light feature map output by a stage sub-model of the first network model as input, it is determined whether the current image was taken during the day or at night, and the feature map output by each stage sub-model in the first network model is adjusted using internal feature representation.

[0087] In some embodiments of this disclosure, the step of using the infrared light feature map output by a stage sub-model of the first network model as input to determine whether the current image was taken during the day or at night, and adjusting the feature map output by each stage sub-model in the first network model using internal feature representations, includes:

[0088] The feature map of the visible light image output by the first stage sub-model is average pooled and then input into two fully connected layers. The internal feature representation is generated by passing through the first activation function.

[0089] Based on the internal feature representation, generate the first and second adjustment parameters for each stage sub-model;

[0090] The feature map of each stage sub-model is adjusted according to the first and second adjustment parameters of each stage sub-model.

[0091] According to another aspect of this disclosure, an apparatus for constructing a target detection model is provided, comprising:

[0092] The first construction module is configured to construct a first network model, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0093] The second construction module is configured to construct a second network model, wherein the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps;

[0094] The third building module is configured to build a third network model, wherein the third network model is used to perform bounding box regression and classification for object detection. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to implement object detection.

[0095] According to another aspect of this disclosure, an object detection model training apparatus is provided, wherein the object detection model training apparatus is configured to train an object detection model using image training data, such that the trained object detection model is used to perform object detection, wherein the object detection model includes a first network model, a second network model, and a third network model.

[0096] The target detection model training device includes:

[0097] The first training module is configured to perform feature extraction and fusion training on the first network model, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0098] The second training module is configured to train the second network model to perform multi-scale feature fusion, wherein the second network model is used to obtain the multi-scale feature map of the fused feature map and perform multi-scale feature fusion on the multi-scale feature map;

[0099] The third training module is configured to train a third network model for object detection. The third network model is used to perform bounding box regression and classification to perform object detection. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to implement object detection.

[0100] According to another aspect of this disclosure, a target detection apparatus is provided, comprising:

[0101] The image receiving module is configured to receive visible light and infrared images at the current moment;

[0102] The target detection module is configured to extract features from the visible light image and the infrared image at the current time, respectively, to obtain feature maps of the visible light image and the infrared image; to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map; and to perform bounding box regression and classification on the fused feature map to perform target detection.

[0103] According to another aspect of this disclosure, a computer apparatus is provided, comprising:

[0104] The memory is configured to store instructions;

[0105] The processor is configured to execute the instructions, causing the computer device to implement the object detection model construction method, the object detection model training method, or the object detection method as described in any of the above embodiments.

[0106] According to another aspect of this disclosure, a target detection system is provided, comprising:

[0107] A visible light camera is configured to capture a visible light image of the current moment.

[0108] An infrared camera is configured to capture infrared images of the current moment.

[0109] The target detection device according to any of the above embodiments.

[0110] In some embodiments of this disclosure, the target is a pedestrian.

[0111] In some embodiments of this disclosure, the visible light camera and the infrared camera are mounted on the top of the vehicle.

[0112] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the object detection model construction method, the object detection model training method, or the object detection method as described in any of the above embodiments.

[0113] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the object detection model construction method, the object detection model training method, or the object detection method as described in any of the above embodiments.

[0114] This disclosure addresses the issue of feature fusion timing between different modalities by employing a multi-level feature fusion scheme to enhance feature interaction between visible light and infrared images, thereby making the fusion between different modalities more thorough. Attached Figure Description

[0115] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0116] Figure 1 These are schematic diagrams illustrating some embodiments of the object detection model construction method disclosed herein.

[0117] Figure 2 This is a schematic diagram of some embodiments of the target detection model disclosed herein.

[0118] Figure 3 This is a network structure diagram of the modality alignment module in some embodiments of this disclosure.

[0119] Figure 4 This is a network structure diagram of the differential information fusion module in some embodiments of this disclosure.

[0120] Figure 5 These are schematic diagrams of other embodiments of the object detection model construction method disclosed herein.

[0121] Figure 6 This is a schematic diagram of some embodiments of the task condition lighting module disclosed herein.

[0122] Figure 7 These are schematic diagrams illustrating some embodiments of the object detection model training method disclosed herein.

[0123] Figure 8These are schematic diagrams of other embodiments of the object detection model training method disclosed herein.

[0124] Figure 9 These are schematic diagrams of some embodiments of the target detection method disclosed herein.

[0125] Figure 10 These are schematic diagrams of some embodiments of the target detection model construction apparatus of this disclosure.

[0126] Figure 11 These are schematic diagrams of some embodiments of the target detection model training device disclosed herein.

[0127] Figure 12 These are schematic diagrams of some embodiments of the target detection device disclosed herein.

[0128] Figure 13 This is a schematic diagram of some embodiments of the computer device disclosed herein.

[0129] Figure 14 These are schematic diagrams of some embodiments of the target detection system disclosed herein. Detailed Implementation

[0130] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0131] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0132] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0133] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0134] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0135] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0136] The inventors also discovered through research that the fusion strategy for the two different modes of infrared and visible light in related technologies is unclear, resulting in minimal fusion benefits. Furthermore, multispectral pedestrian detection networks are typically dual-stream structures, processing RGB image feature extraction and infrared image feature extraction separately, which easily leads to high complexity, poor real-time performance, and limited engineering potential.

[0137] The relevant technology, pedestrian detection method based on visible light cameras, captures images using three color channels: red, green, and blue. However, it performs poorly at night. Nighttime is a period of frequent occurrence for illegal activities, terrorist activities, military operations, and traffic accidents due to insufficient light intensity and poor contrast, making it difficult for humans to clearly observe targets and the environment. The poor nighttime performance of visible light cameras can be mainly attributed to three factors: (a) Light sources are scarce at night, and the uneven spatial distribution of light causes the visible light camera to fail to acquire sufficient light signals, resulting in uneven brightness and thus degrading image quality; (b) The background is darker at night, making background noise (such as smoke and dust) more noticeable, further affecting image quality; (c) Fog and haze are more common at night than during the day, and particulate matter in the air scatters and absorbs light, thus reducing the imaging quality of the visible light camera. This is because temperatures typically drop at night, leading to an increase in the density of water vapor in the air. When water vapor encounters a cooler surface, it condenses into small droplets, forming fog. These water vapors can combine with large amounts of particulate matter to form phenomena such as smog. Therefore, the main drawback of pedestrian detectors based on visible light cameras is that they have difficulty capturing pedestrians at night and are highly sensitive to changes in lighting conditions.

[0138] Similar to other multimodal fusion tasks, the challenges of multispectral pedestrian detection fusion mainly lie in the data space and feature space. Regarding the data space, the problem lies in the significant differences in sensitivity and performance of different modalities to illumination intensity, stemming from the imaging principles of different imaging systems. During the day, pedestrian instances in visible light images have clearer textures than those in infrared images, while at night, due to significant differences in thermal radiation between different types of objects, pedestrians in infrared images are more prominent. Therefore, illumination intensity has a significant impact on pedestrian detection performance, and accurately determining whether an image was captured during the day or night is crucial for robust pedestrian detectors; however, existing multispectral pedestrian detection methods do not consider this. Regarding the feature space, the main issue is that CNNs (Convolutional Neural Networks) prioritize different features extracted from different modalities. Due to differences in imaging principles and physical characteristics between RGB and infrared cameras, the same object exhibits different appearances in the feature maps of different modalities, and its position may also shift.

[0139] Therefore, the challenges of related technologies in the feature space can be summarized into three points: (a) the timing of feature fusion for different modalities: most existing multispectral fusion methods perform fusion at the pixel level, feature level, and decision level, but different fusion timings have different advantages and disadvantages. Pixel-level fusion combines multimodal information in the preprocessing step, which requires high spatiotemporal alignment of the initial data, while feature-level fusion is used in the feature processing layer. Due to the large number of layers in deep convolutional neural networks, it is difficult to determine which feature layer to fuse at. Decision-level fusion focuses on fusing the results of dual-stream networks at the decision layer, which will lose a large number of intermediate semantic features; (b) the fusion strategy for different modalities: since the local receptive fields of CNNs focus on different modal data at different levels, inconsistent and insufficient fusion will lead to an imbalance in the contribution and performance of features. Simple fusion strategies, such as linear combination, summation, concatenation or cascading of features at different levels, lack clear cross-modal complementary extraction. Effective fusion strategies require further exploration. (c) Feature space misalignment problem between different modalities: Due to the gaps between different modalities in the data space (caused by sensor calibration and position), the features extracted in the two modalities are inconsistent, which further hinders the fusion performance and limits the benefits of multimodal fusion.

[0140] Therefore, the inventors have proposed a method and apparatus for constructing an object detection model, a method and apparatus for training an object detection model, an object detection method, apparatus and system, a computer device and a storage medium. These are described below through specific embodiments.

[0141] Figure 1These are schematic diagrams illustrating some embodiments of the object detection model construction method of this disclosure. Preferably, this embodiment can be executed by the object detection model construction apparatus of this disclosure. Figure 1 As shown, the object detection model training method disclosed herein may include steps 1 to 3.

[0142] In step 1, a first network model is constructed, wherein the first network model is used to extract features from the visible light image and infrared image of the target respectively, and to perform multiple feature fusions on the features extracted from the visible light image and infrared image.

[0143] In some embodiments of this disclosure, the first network model is used to extract features from the visible light image and the infrared image of the target, respectively, to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0144] In some embodiments of this disclosure, the target may be a pedestrian.

[0145] In some embodiments of this disclosure, the first network model may be a backbone network, which may be a convolutional neural network that aggregates and forms image features at different image fine-grained levels.

[0146] Figure 2 These are schematic diagrams illustrating some embodiments of the object detection model disclosed herein. Figure 2 As shown, the target detection model may include a first network model 10, a second network model 20, and a third network model 30. The trained target detection model is used to achieve target detection.

[0147] In some embodiments of this disclosure, such as Figure 2 As shown, the first network model includes multiple stage sub-models connected in series. Each stage sub-model may include a visible light extraction sub-model 11, an infrared extraction sub-model 12, and a fusion sub-model 13.

[0148] Visible light extraction sub-model 11 is used to extract features from the visible light image of the target or the feature map of the visible light image output by the previous stage sub-model, so as to obtain the feature map of the visible light image of the current stage sub-model.

[0149] Infrared extraction sub-model 12 is used to extract features from the infrared image of the target or the feature map of the infrared image of the previous stage sub-model to obtain the feature map of the infrared image of the current stage sub-model.

[0150] Fusion sub-model 13 is used to perform feature fusion on the feature maps of the visible light image and the infrared image of the current stage sub-model.

[0151] In some embodiments of this disclosure, such as Figure 2 As shown, step 1 may include: constructing multiple cascaded stage sub-models, wherein the first network model includes n stage sub-models, where n is a natural number greater than 1, and each stage sub-model includes a visible light extraction sub-model and an infrared extraction sub-model. The visible light extraction sub-model 11 and the infrared extraction sub-model 12 of the first stage sub-model are used to perform convolution operations and downsampling on the input visible light image and infrared image to generate the feature map of the first stage; the visible light extraction sub-model 11 and the infrared extraction sub-model 12 of the i-th stage sub-model are used to perform convolution operations and downsampling on the feature map generated by the (i-1)-th stage sub-model to generate the feature map of the i-th stage, where i is a natural number greater than 1 and less than or equal to n.

[0152] In some embodiments of this disclosure, such as Figure 2 As shown, the first network model includes 5 stage sub-models (stage 1 to stage 5). Each stage sub-model includes a visible light extraction sub-model and an infrared extraction sub-model. The first stage sub-model (stage 1) performs convolution operations and downsampling on the input visible light image and infrared image. The i-th stage sub-model performs convolution operations and downsampling on the feature map of the (i-1)-th stage sub-model, where i is a natural number greater than 1 and less than or equal to 5.

[0153] In some embodiments of this disclosure, such as Figure 2 As shown, in the first network model 10 (Backbone network) involving multiple fusions: Since the same objects, such as pedestrians and vehicles, exhibit different feature representations in visible light and infrared images, and as the number of CNN (Convolutional Neural Networks) layers increases, deeper features become more abstract, possessing higher-level semantic information but poorer localization and detail information. Therefore, the choice of which feature layer to fuse is crucial. Furthermore, during feature extraction, downsampling operations generate a feature pyramid network.

[0154] In some embodiments of this disclosure, such as Figure 2 As shown, in the first network model 10 (Backbone network) with multiple fusions, four fusions were performed to fully integrate the features of different modalities. Each fusion adopted a cascaded structure of MAM (Modality Alignment Module) and DMFM (Differential Modality Fusion Module). MAM is a learning-based module that can adaptively align the feature space. Then, DMFM utilizes the differences in modal information to enhance the relationship between different modalities, suppress common modalities, and amplify differential modalities.

[0155] In some embodiments of this disclosure, such as Figure 2 As shown, in order to fully fuse feature maps of different network depths and modalities, this disclosure adopts the idea of ​​a two-stream feature extraction network and multiple feature fusion. Specifically, the first network model 10 (e.g., the Backbone network) is divided into 5 stages, namely Stage 1 to Stage 5, as follows: Figure 2 As shown. Each stage performs a series of convolutional operations and downsampling on the feature map of the previous stage, thereby generating a feature map with smaller resolution, more abstract, and higher dimension. Since the Neck structure follows Stage 5, it is necessary to integrate the dual-stream feature extraction networks of different modalities into a single-stream network. Therefore, Stage 5 adopts a different fusion method than Stages 1 to 4. In detail, in Stages 1 to 4, this disclosure uses a cascaded structure of MAM and DMFM to perform feature space alignment and fusion for different modalities, while Stage 5 directly concatenates the feature maps of different modalities and adjusts the weights of the feature maps by combining the attention map generated by the TCIM (Task-Conditioned Illumination Module, specifically a day / night classification-driven task conditional illumination module) structure. The detailed description of the MAM, DMFM, and TCIM modules is as follows.

[0156] In some embodiments of this disclosure, the step of constructing a series of multiple stage sub-models may include at least one of steps 11 to 14.

[0157] In step 11, multiple modality alignment modules are constructed. The fusion sub-model 13 includes a modality alignment module, which is connected to the visible light extraction sub-model and the infrared extraction sub-model, respectively. The modality alignment module is used to perform feature space alignment on the feature maps of the visible light image and the infrared image.

[0158] In some embodiments of this disclosure, the j-th stage sub-model includes a modal alignment module, where j is a natural number greater than 0 and less than n. The modal alignment module is used to align the feature spaces of different modes, which are visible light modes and infrared modes.

[0159] In some embodiments of this disclosure, such as Figure 2 As shown, a modal alignment module (MAM) is set in the sub-models from the first stage to the fourth stage; the fused sub-model 13 of the sub-models from the first stage to the fourth stage includes the modal alignment module.

[0160] The inventors discovered through research that due to gaps in the data space between visible light and infrared images (caused by sensor calibration errors), the features extracted from the two modalities are inconsistent, which further hinders fusion performance and limits the benefits of multimodal fusion. Therefore, this disclosure modifies the feature spaces of different modalities to enable better fusion of visible light and infrared images, improving the accuracy of multispectral pedestrian detection. However, a major challenge of feature space adaptation is the lack of clear and complete knowledge regarding the conversion between infrared and RGB images, as the differences between them are due to different imaging principles. To address this issue, this disclosure proposes a learning-based modal alignment module to adaptively align the feature space, such as... Figure 3 As shown.

[0161] Figure 3 This is a network structure diagram of the modality alignment module in some embodiments of this disclosure. For example... Figure 3 As shown, the modality alignment module (MAM) may include a feature concatenation model 31, a first transformation network (TransNet1) 32, a second transformation network (TransNet2) 33, a feature space adaptation operation model 34, and a convolutional model 35. Figure 3 As shown. Figure 3 In this context, 3×3Comv represents a 2D convolution operation with a kernel size of 3, and 1×1Conv represents a 2D convolution operation with a kernel size of 1.

[0162] Feature cascading model 31 is used to cascade the feature maps of visible light images and infrared images to generate cascaded feature maps.

[0163] The first transformation network (TransNet1) 32 is used to process the input cascaded feature map to obtain a first scaling factor and a first offset, wherein the first scaling factor and the first offset are the scaling factor and offset in the feature space transformation relationship (TransNet1) from infrared image to visible light image.

[0164] The second transformation network (TransNet2) 33 is used to process the input cascaded feature map to obtain a second scaling factor and a second offset, wherein the second scaling factor and the second offset are the scaling factor and offset in the feature space transformation relationship (TransNet2) from visible light image to infrared image.

[0165] In some embodiments of this disclosure, a feature space adaptation operation can be defined using a linear transformation, as shown in Equation (1).

[0166] f adapted =α ⊙ f orgin ⊕β (1)

[0167] In formula (1), forgin f represents the original feature space. adapted Let F represent the adaptively transformed feature space, where α and β represent scaling factors and offsets, respectively. All four tensors have the shape (W, H, C), where C refers to Channel (number of channels), H is the height (vertical coordinate), and W is the width (horizontal coordinate). ⊙ represents element-wise multiplication, and ⊕ represents element-wise summation. For multispectral fusion scenarios, it is necessary to learn both the feature space transformation relationship from visible light to infrared images and from infrared to visible light images, thus requiring the learning of two pairs of α (scaling factors) and β (offsets). To accurately estimate α and β, this disclosure introduces a TransNet to learn and adapt multispectral features, and converts the RGB feature map F... R and Thermal feature map F T The cascaded feature maps are used as input to the transformation network, F R and F T The shapes are all (W,H,C), as shown in formula (2).

[0168] f concat =Concat(F R ,F T )

[0169] (α1,β1)=TranSNet1(f concat (2)

[0170] (α2,β2)=TransNet2(f concat )

[0171] In formula (2), TransNet1 and TransNet2 are both two parallel 2D convolution operations with a kernel size of 1, respectively applied to f concat This generates the corresponding scaling factor and offset. Since the number of channels doubles after concatenation, to ensure that the scaling factor and offset are the same as the original feature map's channel number, the output of this 2D convolution will be f... concat The number of channels is reduced by half. In addition, TransNet1 and TransNet2 do not share weights: α1 (first scaling factor) and β1 (first offset) for adapting to the visible light feature space, and α2 (first scaling factor) and β2 (second offset) for adapting to the infrared feature space.

[0172] In formula (2), `concat` means concatenate. In deep learning, concatenate is a common operation used to merge multiple feature vectors or feature maps into a single vector or map. This process usually occurs between different layers of a neural network or in the feature fusion layer of the model. The main purpose of concatenate is to increase the expressive power of the model, enabling the network to learn richer feature representations. For vector concatenation: if the input is two or more vectors of the same dimension, concatenate will connect them along the same dimension to form a new vector. For example, if there are two vectors with dimensions [1, 10], the new vector obtained after concatenation will have dimensions [1, 20]. For feature map concatenation: in convolutional neural networks, feature maps are two-dimensional arrays output by convolutional layers. Concatenate can merge these feature maps along the depth (channel) dimension, thereby increasing the depth of the feature maps. For example, if there are two feature maps with depths [1, 10, 10], the new feature map obtained after concatenation will have depths [1, 10, 20].

[0173] The feature space adaptation operation model 34 is used to perform element-wise multiplication of the feature map of the infrared image with the first scaling factor, and then element-wise summation with the first offset to obtain the feature map of the adaptively transformed infrared image; and to perform element-wise multiplication of the feature map of the visible light image with the second scaling factor, and then element-wise summation with the second offset to obtain the feature map of the adaptively transformed visible light image, such as... Figure 3 As shown.

[0174] The convolution model is used to sum the feature maps of the visible light image and the adaptively transformed infrared image element-wise, and then perform a convolution operation to obtain the transfer feature map of the visible light image; similarly, it is used to sum the feature maps of the infrared image and the adaptively transformed visible light image element-wise, and then perform a convolution operation to obtain the transfer feature map of the infrared image.

[0175] In some embodiments of this disclosure, such as Figure 3 As shown, the updated feature map of the Modal Alignment Module (MAM) of this disclosure is shown in Equation (3).

[0176]

[0177] In formula (3), and These represent the feature maps output by MAM, where... This represents the transfer feature map of the visible light image (RGB) after MAM output. and These represent the feature maps of the transition feature map after MAM output and the infrared image (thermal), respectively. 3×3 This represents a 2D convolution operation with a kernel size of 3, used to eliminate aliasing of feature values ​​caused by the superposition of feature maps from different modalities.

[0178] By optimizing the learnable parameters of the MAM, this disclosure enables visible light and infrared image features to learn appropriate feature space transformations from each other, thereby promisingly improving person detection performance. Furthermore, to preserve detail information in the original feature maps, this disclosure performs pixel-level summation between the original feature maps and the MAM-processed feature maps, using this sum as input to subsequent modules. Compared to overall network architectures that may have dozens of convolutional layers, the complexity of the MAM is relatively low. In summary, the essence of the MAM disclosed in this disclosure is to promote feature space alignment between different modalities, enabling one modality to adaptively adjust its feature space according to another modality.

[0179] This disclosure proposes a learning-based modal alignment module (MAM) to adaptively align feature spaces. It utilizes linear transformations to define the feature space adaptation operation, including the mutual conversion between infrared image feature spaces and visible light image feature spaces. Furthermore, to preserve detail information in the original feature maps, this disclosure performs pixel-level summation between the original feature maps and the MAM-processed feature maps, using this summation as input to subsequent modules.

[0180] In step 12, multiple differential information fusion modules are constructed. The j-th stage sub-model includes a differential information fusion module, which is used to enhance the relationship between one mode and another mode by utilizing the differences in modal information. The mode is a visible light mode or an infrared mode.

[0181] In some embodiments of this disclosure, such as Figure 2 As shown, the fusion sub-model also includes a differential information fusion module (DMFM), which is connected to the visible light extraction sub-model and the infrared extraction sub-model respectively. The differential information fusion module is used to enhance the relationship between the feature maps of the visible light image and the infrared image by utilizing the differences between the feature maps of the visible light image and the infrared image.

[0182] In some embodiments of this disclosure, inconsistent and insufficient fusion can lead to an imbalance in feature contribution and performance because the local receptive fields of CNNs focus on different modalities differently. Simple fusion strategies, such as linear combination, summation, concatenation or cascading of features at different levels, lack clear cross-modal complementary extraction and mix useful information with redundant noise. To address the problem of feature modality imbalance, inspired by differential amplifier circuits, this application proposes a Differential Information Fusion (DMFM) module, which utilizes the differences in modal information to enhance the relationship between one modality and another. The common modality reflects the shared features captured by the two modalities, while the differential modality partially reflects the unique features captured by the two modalities. The feature map F of the visible light image... R Feature map F of infrared image T It can be represented by common modes and differential modes, as shown in formula (4):

[0183]

[0184] In formula (4), the common modes are all present in the feature maps of visible light images and infrared images, so there is no need to consider them too much. Instead, the focus should be on the difference information between the two modes.

[0185] The key idea of ​​this DMFM module is to differentially weight the channel features of different modalities, then obtain complementary features from another modality, and explicitly model the interdependencies between different modalities to enhance the learning of complementary features. This improves the network's sensitivity to information features from another modality. The network structure is as follows: Figure 4 As shown.

[0186] Figure 4 This is a network structure diagram of the differential information fusion module in some embodiments of this disclosure. For example... Figure 4 As shown, the differential information fusion module of this disclosure may include a differential feature determination model 41, an auxiliary feature determination model 42, and a differential fusion model 43.

[0187] The differential feature determination model 41 is used to determine the first differential feature F based on the difference between the feature map of the infrared image and the feature map of the visible light image. T -F R Based on the difference between the feature maps of the visible light image and the infrared image, the second difference feature F is determined. R -F T ,like Figure 4 As shown.

[0188] In some embodiments of this disclosure, the differential feature determination model 41 is used to subtract two feature maps from different modalities to obtain differential features F. T -F R and FR -F T ,like Figure 4 As shown.

[0189] The differential feature determination model 42 is used to calculate a feature map to assist the visible light image based on the first differential feature, and to calculate a feature map to assist the infrared image based on the second differential feature.

[0190] In some embodiments of this disclosure, such as Figure 2 As shown, the differential feature determination model 42 may include a weight factor generation module 421 and an auxiliary feature determination module 422.

[0191] The weight factor generation module 421 is used to perform global average pooling on the first difference feature, perform channel scaling and restoration through convolution, and generate a first weight factor through an activation function; and to perform global average pooling on the second difference feature, perform channel scaling and restoration through convolution, and generate a second weight factor through an activation function.

[0192] The auxiliary feature determination module 422 is used to perform element-level feature integration between the first weight factor and the feature map of the infrared image to obtain a feature map used to assist the visible light image; and to perform element-level feature integration between the second weight factor and the feature map of the visible light image to obtain a feature map used to assist the infrared image.

[0193] Differential fusion model 43 is used to sum the feature map of the visible light image with the feature map used to assist the visible light image at the element level to determine the fused feature map of the visible light image, and to sum the feature map of the infrared image with the feature map used to assist the infrared image at the element level to determine the fused feature map of the infrared image.

[0194] In some embodiments of this disclosure, such as Figure 4 As shown, the differential information fusion module disclosed herein is used to subtract feature maps from two different modalities to obtain differential features F. T -F R and F R -F TThen, the differential features are compressed into a global differential vector through global average pooling (GAP). Its statistical data represents the feature difference between RGB and infrared images. After two 1×1 convolutions (1×1 Conv), the channels are scaled and restored. This process can capture the nonlinear changes between channels, establish the interdependence between channels, and thus better update the channel weights. Finally, the activation function of Tanh (hyperbolic tangent function) is used to generate a weight factor between -1 and 1. Then, element-wise multiplication is performed with the original input feature map, and element-wise feature summation is performed with the original features of another modality. The detailed process is shown in formula (5).

[0195]

[0196] In formula (5), and F represents the final output of DMFM. T_to_R Indicates the difference features (F) T -F R The feature map F, calculated in F, is used to assist the RGB mode. R_to_T Indicates the difference features (F) R -F T The feature map calculated in () is used to assist the infrared mode, where ( represents the Tanh activation function). Furthermore, to preserve detailed information in the original feature map, such as Figure 2 As shown, this disclosure performs pixel-level summation between the original feature map and the feature map processed by DMFM, and uses this summation as input for subsequent modules.

[0197] This disclosure proposes a Differential Information Fusion Module (DMFM) that leverages the differences in modal information to enhance the relationship between one modality and another. Specifically, DMFM enhances the learning of complementary features by differentially weighting the channel features of different modalities, then extracting complementary features from another modality, and explicitly modeling the interdependencies between different modalities. This improves the network's sensitivity to information features from another modality.

[0198] In step 13, a stitching module is constructed, wherein the first network model includes n stage sub-models, where n is a natural number greater than 1, and the fusion sub-model 13 of the nth stage sub-model includes the stitching module, which is used to stitch together the feature maps of the visible light image and the infrared image generated by the (n-1)th stage sub-model.

[0199] In some embodiments of this disclosure, such as Figure 2As shown, the fifth stage sub-model (stage 5) includes the stitching module, which is used to stitch together the feature maps of the visible light image and the infrared image.

[0200] In step 14, an input determination module is constructed, wherein the input determination module is used to determine that the input of the visible light extraction sub-model in the i-th stage sub-model is the sum of the original input of the visible light extraction sub-model, the transfer feature map of the visible light image after passing through the modal alignment module of the (i-1)-th stage sub-model, and the fusion feature map of the visible light image after passing through the differential information fusion module of the (i-1)-th stage sub-model; and to determine that the input of the infrared extraction sub-model in the i-th stage sub-model is the sum of the original input of the infrared extraction sub-model, the transfer feature map of the infrared image after passing through the modal alignment module of the (i-1)-th stage sub-model, and the fusion feature map of the infrared image after passing through the differential information fusion module of the (i-1)-th stage sub-model, such as... Figure 2 As shown, the first network model includes n stage sub-models, where n is a natural number greater than 1 and i is a natural number greater than 1 and less than or equal to n.

[0201] In some embodiments of this disclosure, such as Figure 2 As shown, MAM and DMFM are parallel, and the inputs to both MAM and DMFM are multimodal features F. R and F R .

[0202] In some embodiments of this disclosure, such as Figure 2 As shown, the combination of MAM and DMFM is a cross structure.

[0203] In some embodiments of this disclosure, such as Figure 2 As shown, the subsequent input of the RGB branch is equal to the feature map after MAM (the transferred feature map of the visible light image after the modality alignment module) + the feature map after DMFM (the fused feature map of the visible light image after the differential information fusion module) + the original input of the RGB branch.

[0204] In step 2, a second network model is constructed, wherein the second network model is used to perform multi-scale feature fusion on the image features output by the first network model.

[0205] In some embodiments of this disclosure, the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps.

[0206] In some embodiments of this disclosure, the second network model may be a neck network, comprising a series of network layers that mix and combine image features to pass the image features to the prediction layer. The main function of the second network model is to reduce or adjust the features from the backbone (first network model) to better suit the task requirements.

[0207] In step 3, a third network model is constructed, which is used to perform bounding box regression and classification, target detection, and prediction of the image features of the target.

[0208] In some embodiments of this disclosure, the third network model may be a Head network. The third network model is used to predict image features, generate bounding boxes, and predict categories.

[0209] Figure 5 These are schematic diagrams illustrating other embodiments of the object detection model construction method of this disclosure. Preferably, this embodiment can be executed by the object detection model construction apparatus of this disclosure. Figure 5 As shown, the object detection model training method of this disclosure may include at least one of steps 1 to 4, wherein, Figure 5 Steps 1-3 of the embodiment and Figure 1 Steps 1-3 of the embodiment are the same or similar.

[0210] In step 1, a first network model is constructed, wherein the first network model is used to extract features from the visible light image and infrared image of the target respectively, and to perform multiple feature fusions on the features extracted from the visible light image and infrared image.

[0211] In some embodiments of this disclosure, the first network model is used to extract features from the visible light image and the infrared image of the target, respectively, to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0212] In step 2, a second network model is constructed, wherein the second network model is used to perform multi-scale feature fusion on the image features output by the first network model.

[0213] In some embodiments of this disclosure, the second network model is a Neck network structure that integrates multi-scale features.

[0214] In some embodiments of this disclosure, step 2 may include: constructing a top-down structure, wherein the top-down structure is used to upsample the feature map output by the first network model, and merge the upsampled feature map with the feature maps output by the visible light extraction sub-model and the infrared extraction sub-model in the first network model through lateral connection after feature extraction; constructing a bottom-up structure, wherein the bottom-up structure is used to stitch together multi-layer feature maps, wherein the multi-layer feature maps are feature maps processed by the top-down structure at different times.

[0215] In some embodiments of this disclosure, such as Figure 2 As shown, the second network model (Neck) network structure includes two parts: top-down and bottom-up. As the neural network deepens, the extracted features become more abstract, possessing high-level semantic information, but simultaneously losing rich details and positional information. Generally, low-level features contain more detailed information, aiding in localization and are crucial for small object detection, while high-level semantic information is important for classification and recognition. In this disclosure, during the Backbone feature extraction process, a Backbone feature pyramid (composed of multiple feature maps of different resolutions) is automatically constructed due to the multiple downsampling methods employed. The Top-down method of this disclosure spatially upsamples the low-resolution feature map output by Backbone by a factor of 2 (nearest neighbor interpolation). Then, the upsampled feature map, after feature extraction, is merged with the Backbone feature map through lateral connections (passing through a 1×1 convolutional layer to reduce the number of channels), as shown below. Figure 2 As shown, merging via lateral joins refers to merging the feature map after the first upsampling, after feature extraction, with the feature map of stage 4 in the Backbone via lateral joins; and merging the feature map after the first upsampling, after feature extraction, with the feature map of stage 3 in the Backbone via lateral joins. This process is repeated until the highest resolution map is generated. Following the Top-down structure, a Bottom-up structure is further applied to enhance the feature representations at each level, shorten the information path between low-level and high-level features, strengthen the flow of feature information, and fully aggregate features from different layers, thereby significantly improving detection performance.

[0216] In some embodiments of this disclosure, such as Figure 2 As shown, in the second network model (Neck), Top-down refers to generating new outputs by connecting lines from Stage 3 and Stage 4 to the Neck. Bottom-up refers to... Figure 2The concatenation part of the Neck concatenates the top-down result with the original feature map in the Neck to form a bottom-up structure, and then the bottom-up output is sent to the head (third network model).

[0217] In step 3, a third network model is constructed, which is used to perform bounding box regression and classification, target detection, and prediction of the image features of the target.

[0218] In some embodiments of this disclosure, the third network model may be a Head network structure used to perform bounding box regression and classification.

[0219] In some embodiments of this disclosure, step 3 may include: constructing multiple detection heads of different sizes, wherein the multiple detection heads of different sizes are used to cover targets of different scales, and each detection head is used to predict whether a target exists within the grid and to determine target information, wherein the target information includes the target's location, size, and category.

[0220] In some embodiments of this disclosure, for training, the fusion network uses GIoU (Generalized Intersection over Union) loss as the bounding box regression loss and binary cross-entropy loss as the classification prediction loss. To improve the detection accuracy of targets at different scales, the fusion network uses three detector heads of different sizes to cover targets at different scales: large-scale, medium-scale, and small-scale. The network is divided according to the original image corresponding to the detector heads of different scales. Each detector head is responsible for predicting whether a target exists within the grid, as well as the target's location, size, and category. Generally speaking, the larger the feature map, the smaller the receptive field, which is more suitable for detecting small targets. By cleverly cascading the three detector heads, the detection accuracy and recall can be significantly improved. In addition, the fusion network also uses weighted non-maximum suppression to remove duplicate bounding boxes based on the detection confidence score, thereby improving the detection performance of occluded targets.

[0221] In some embodiments of this disclosure, the image feature extractor and all modules of this disclosure employ convolutional neural networks. Alternatively, the image feature extractor and all modules of this disclosure may employ encoders of Vision Transformer variants or convolutional neural network variants.

[0222] In step 4, a fourth network model is constructed, which is connected to the first network model. The fourth network model is used to determine whether the current image was taken during the day or at night by using the feature map output by the visible light extraction sub-model of the first network model as input, and to adjust the feature map output by each stage sub-model in the first network model by using internal feature representation.

[0223] In some embodiments of this disclosure, such as Figure 2 As shown, the fourth network model is used to determine whether the current image was taken during the day or at night by using the feature map of the first stage of the first network model as input, and to adjust the feature map of the first network model by using the internal feature representation.

[0224] In some embodiments of this disclosure, such as Figure 2 As shown, the target detection model may include a fourth network model 40.

[0225] In some embodiments of this disclosure, such as Figure 2 As shown, the fourth network model of this disclosure can be a day / night classification-driven Task Conditional Illumination Module (TCIM).

[0226] Because visible light images and infrared images exhibit different characteristics in response to light intensity—visible light images show clearer texture features during the day, while infrared images show more distinct pedestrian shapes at night—determining whether an image was captured during the day or at night is crucial for robust pedestrian detectors. Therefore, this disclosure proposes a TCIM structure, such as... Figure 6 As shown. Figure 6 This is a schematic diagram of some embodiments of the Task Conditional Illumination Module (TCIM) of this disclosure. TCIM uses the input of a low-level feature-assisted classification network, leveraging its learned internal feature representations to capture useful salient information to determine whether an image was captured during the day or at night, thereby generating an attention map to adjust the weights of the pedestrian detection network and improving the accuracy of multispectral pedestrian detection.

[0227] In some embodiments of this disclosure, such as Figure 2 and Figure 6 As shown, this disclosure defines a pedestrian detection main network and an auxiliary classification network. The pedestrian detection main network includes a first network model 10, a second network model 20, and a third network model 30, which are... Figure 2 and Figure 6 The lower half consists of branches of the first network model 10, the second network model 20, and the third network model 30; the auxiliary classification network is the fourth network model 40. Figure 2 and Figure 6 The upper part is the auxiliary binary classification network branch.

[0228] In some embodiments of this disclosure, step 4 may include at least one of steps 41 to 43.

[0229] In step 41, a feature representation generation module is constructed. The feature representation generation module is used to perform average pooling on the feature map of the visible light image of the visible light extraction sub-model in the first stage sub-model, and then input it into two fully connected layers. Through the first activation function, an internal feature representation is generated.

[0230] In some embodiments of this disclosure, the first activation function may be a ReLU activation function.

[0231] In some embodiments of this disclosure, feature maps from shallower networks are used as input because the earlier feature maps of the CNN contain more detailed information (e.g., ...). Figure 2 and Figure 6 In stage 1 of the classification process, this is of great significance for distinguishing between day and night. Infrared images are less sensitive to light intensity than visible light images; therefore, this disclosure uses feature maps from visible light images as input to an auxiliary classification network.

[0232] In some embodiments of this disclosure, such as Figure 6 As shown, the feature representation generation module is used to perform average pooling on the early feature map of the visible light image, and then input it into two fully connected layers (FC) with a feature dimension of size C. Subsequently, the ReLU activation function is used to generate the internal feature representation.

[0233] In some embodiments of this disclosure, such as Figure 6 As shown, the task-conditional illumination module can also be used to pass the generated internal feature representation to a fully connected layer (2FC) with 2 channels, followed by a second activation function (e.g., a sigmoid activation function) to classify day and night. To train the auxiliary classification network, this disclosure associates an auxiliary training label with all training images. Since the fusion network was experimented on the KAIST and CVC-14 datasets, which only distinguish between day and night images in their annotations and do not provide quantified illumination intensity metrics, this disclosure labels night images as 0 and day images as 1, and employs a binary cross-entropy loss function. In this case, the task of the auxiliary network is to classify the image as day or night.

[0234] In step 42, an adjustment parameter generation module is constructed, wherein the adjustment parameter generation module is used to generate the first adjustment parameter and the second adjustment parameter of each stage sub-model based on the internal feature representation.

[0235] In some embodiments of this disclosure, the adjustment parameter generation module may include a first adjustment parameter generation submodule and a second adjustment parameter generation submodule, wherein: the first adjustment parameter generation submodule is used to determine the weights and biases corresponding to the first adjustment parameter of the k-th stage sub-model; and to determine the first adjustment parameter based on the weights and biases corresponding to the first adjustment parameter, the internal feature representation, and the first activation function; the second adjustment parameter generation submodule is used to determine the weights and biases corresponding to the second adjustment parameter of the k-th stage sub-model; and to determine the second adjustment parameter based on the weights and biases corresponding to the second adjustment parameter, the internal feature representation, and the first activation function.

[0236] In step 43, a feature map adjustment module is constructed, wherein the feature map adjustment module is used to adjust the feature map of each stage sub-model according to the first adjustment parameter and the second adjustment parameter of each stage sub-model.

[0237] In some embodiments of this disclosure, the feature map adjustment module may include a third adjustment parameter generation submodule and a feature map adjustment submodule. The third adjustment parameter generation submodule is used to determine a third adjustment parameter based on the difference between 1 and the first adjustment parameter. The feature map adjustment submodule is used to multiply the third adjustment parameter with the feature map of the k-th stage sub-model, then sum it with the second adjustment parameter, and input it into the first activation function to obtain the adjusted feature map of the k-th stage sub-model.

[0238] This disclosure modulates the convolutional feature map by using the internal feature representations of an auxiliary classification network (after the two fully connected layers used for classification) to generate tuning parameters. These internal feature representations learn salient information to determine whether the image was captured during the day or at night. Figure 6 As shown, the adjustment parameters can be embedded into the feature maps after any stage. TCIM will adjust the visible light and infrared feature maps of stages 1-5.

[0239] In the final experiment, this disclosure demonstrates that inserting the feature maps in all stages (visible and infrared features of stages 1-5) yields the best results. Equation (6) represents the calculation process of the adjustment layer feature maps.

[0240]

[0241] In formula (6), k represents the feature map of the k-th stage, and A(x) represents the internal feature representation of the auxiliary classification network. This represents the weight corresponding to γ ​​in the k-th stage. This represents the bias corresponding to γ ​​in the k-th stage. and Similarly, ReLU represents the activation function.

[0242] Formula (7) represents the adjustment of the Backbone feature map, which can guide the fusion of multispectral features by deriving the feature map before classification.

[0243] F : / (4)=Re01[(1-. / )⊙F / (4)⊕7 / (7)

[0244] In formula (7), F / (4) Represents the feature map of the k-th stage of the pedestrian detection main network, F : / (4) represents the updated feature map after parameter adjustment. Therefore, TCIM can improve pedestrian detection performance by dividing the image into day / night segments and then adjusting the pedestrian detection main network through the internal feature representation of the auxiliary classification network, so that the pedestrian detection main network can adaptively adjust the feature map weights during the day and night.

[0245] In some embodiments of this disclosure, such as Figure 2 and Figure 6 As shown, the main network for pedestrian detection includes a first network model (Backbone) 10, a second network model (Neck) 20, and a third network model (Head) 30. A fourth network model (TCIM) 40 is inserted into the Backbone five times: after Stage 1, Stage 2, Stage 3, Stage 4, and Stage 5. Stages 1 through 4 are dual-stream networks, requiring insertion into both the RGB and infrared feature maps simultaneously. Stage 5 integrates the dual-stream networks into a single-stream network through concatenation before inserting TCIM. Figure 2 The insertion shown was performed after Stage 5.

[0246] In some embodiments of this disclosure, such as Figure 2 As shown, the first network model (Backbone) performs thorough feature extraction from visible light and infrared images, and uses MAM and DMFM for feature alignment and fusion respectively. Then, TCIM is used to adjust the feature map weights using feature representations for auxiliary classification, resulting in rich multispectral features. Next, the Backbone's output is fused with multi-scale features through the second network model (Neck), and bounding box regression and classification are performed through the third network model (Head). The following section combines... Figure 2 The network structures of the Neck and Head are described.

[0247] This disclosure proposes a day / night classification-driven Task Conditional Illumination Module (TCIM), which uses low-level features to assist the input of a classification network and leverages its learned internal feature representations to capture useful salient information to determine whether an image was captured during the day or at night. This generates an attention map to adjust the weights of the pedestrian detection network, thereby improving the accuracy of multispectral pedestrian detection.

[0248] Figure 7 These are schematic diagrams illustrating some embodiments of the object detection model training method of this disclosure. Preferably, this embodiment can be executed by the object detection model training apparatus of this disclosure. Figure 7 As shown, the object detection model training method of this disclosure may include step 700.

[0249] In step 700, the target detection model is trained using image training data, so that the trained target detection model can be used to perform target detection. The target detection model includes a first network model, a second network model, a third network model, and a fourth network model.

[0250] In some embodiments of this disclosure, step 700 may include at least one of steps 710 to 740.

[0251] In step 710, feature extraction and fusion training are performed on the first network model. The first network model is used to extract features from the visible light image and infrared image of the target respectively to obtain feature maps of the visible light image and infrared image. Multi-level feature fusion is performed on the feature maps of the visible light image and infrared image to obtain a fused feature map.

[0252] In step 720, the second network model is trained to perform multi-scale feature fusion, wherein the second network model is used to obtain the multi-scale feature map of the fused feature map and perform multi-scale feature fusion on the multi-scale feature map.

[0253] In step 730, the third network model is trained for object detection. The third network model is used to perform bounding box regression and classification, perform object detection, and predict the image features of the object. The object detection model includes the first network model, the second network model, and the third network model. The trained object detection model is used to achieve object detection.

[0254] In step 740, the fourth network model is trained for auxiliary classification. The fourth network model is used to determine whether the current image was taken during the day or at night by using the feature map of the first stage of the first network model as input, and to adjust the feature map of the first network model by using internal feature representation.

[0255] In some embodiments of this disclosure, the target detection model is constructed using the target detection model construction method described in any of the above embodiments.

[0256] Figure 8 These are schematic diagrams illustrating other embodiments of the object detection model training method of this disclosure. Preferably, this embodiment can be executed by the object detection model training apparatus of this disclosure. Figure 8 As shown, the target detection model training method of this disclosure may include at least one of steps 400 to 700.

[0257] In step 400, visible light and infrared images are acquired at the same time and from the same image acquisition location.

[0258] In some embodiments of this disclosure, step 400 may include: acquiring visible light and infrared images from the same time and image acquisition location from the KAIST and CVC-14 datasets. The KAIST pedestrian dataset comprises a total of 95,328 images, each containing both RGB color and infrared modalities. It contains a total of 103,128 dense annotations. The dataset captured various routine traffic scenes, including campuses, streets, and rural areas, both during the day and at night. The CVC-14 dataset contains grayscale and infrared images, with 7,085 frames in the training set and 1,433 frames in the test set. Annotations for different modalities are provided independently due to poor camera alignment.

[0259] In step 500, the visible light image and the infrared image are preprocessed.

[0260] In some embodiments of this disclosure, step 500 may include: first, performing data cleaning to remove the impact of abnormal and duplicate data on model training. Then, adjusting the image size and performing normalization to reduce computational complexity, save memory space, and improve model stability and convergence speed. Next, employing various data augmentation techniques to increase data diversity, such as cropping, rotation, scaling, translation, brightness adjustment, contrast adjustment, hue adjustment, saturation adjustment, and noise addition. Finally, inputting the augmented RGB and infrared images into the designed model.

[0261] In step 600, the preprocessed visible light image and infrared image are used as the image training data.

[0262] In step 700, the target detection model is trained using image training data, so that the trained target detection model can be used to perform target detection. The target detection model includes a first network model, a second network model, a third network model, and a fourth network model.

[0263] In some embodiments of this disclosure, step 700 may further include at least one of steps 750 to 780.

[0264] In step 740, the binary cross-entropy loss function is used as the auxiliary classification loss function for the fourth network model.

[0265] In step 760, the generalized intersection-over-union loss function is used as the bounding box regression loss function in the third network model.

[0266] In step 770, the binary cross-entropy loss function is used as the classification prediction loss function for the third network model.

[0267] In step 780, the loss function of the target detection model is determined based on the auxiliary classification loss function, the bounding box regression loss function, and the classification prediction loss function.

[0268] In step 790, the parameters of the first network model, the second network model, the third network model, and the fourth network model are adjusted according to the loss function of the target detection model.

[0269] Figure 9 This is a schematic diagram of some embodiments of the target detection method of this disclosure. Preferably, this embodiment may be based on the target detection device or the target detection model of this disclosure (e.g., Figures 2 to 4 , Figure 6 The target detection model in the example is executed. Figure 9 As shown, the target detection model training method of this disclosure may include at least one of steps 91 to 94.

[0270] In step 91, the visible light image (RGB image) and infrared image at the current moment are received.

[0271] In some embodiments of this disclosure, step 91 may include at least one of steps 911 and 912.

[0272] In step 911, image data synchronously acquired by a visible light camera and an infrared camera located on the roof of the vehicle is received.

[0273] In some embodiments of this disclosure, the target may be a pedestrian.

[0274] In some embodiments of this disclosure, the target detection device can be implemented as a server.

[0275] In some embodiments of this disclosure, step 911 may include: during the vehicle's operation, a visible light camera and an infrared camera located on the roof of the vehicle simultaneously acquire image data and upload it to a server. To improve the alignment of the two different modal images, this disclosure accurately calibrates the intrinsic and extrinsic parameters of the cameras and ensures that the hardware settings of the two cameras are synchronized, including exposure time, frame rate, and sensitivity.

[0276] In step 912, the visible light image and infrared image at the current moment are preprocessed.

[0277] In some embodiments of this disclosure, step 912 may include: first, performing data cleaning to remove the impact of abnormal and duplicate data on model training. Then, adjusting the image size and performing normalization to reduce computational complexity, save memory space, and improve model stability and convergence speed. Next, employing various data augmentation techniques to increase data diversity, such as cropping, rotation, scaling, translation, brightness adjustment, contrast adjustment, hue adjustment, saturation adjustment, and noise addition. Finally, inputting the augmented RGB and infrared images into the designed model.

[0278] In step 92, feature extraction is performed on the visible light image and the infrared image at the current moment to obtain feature maps of the visible light image and the infrared image, respectively.

[0279] In step 93, multi-level feature fusion is performed on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0280] In some embodiments of this disclosure, step 93, which involves multi-level feature fusion of the feature maps of the visible light image and the infrared image, may include: feature space alignment of the feature maps of the visible light image and the infrared image.

[0281] In some embodiments of this disclosure, the step of aligning the feature maps of the visible light image and the infrared image in a feature space includes: concatenating the feature maps of the visible light image and the infrared image to generate a concatenated feature map; processing the concatenated feature map to obtain a first scaling factor and a first offset, wherein the first scaling factor and the first offset are the scaling factor and the offset in the feature space transformation relationship from the infrared image to the visible light image; and processing the concatenated feature map to obtain a second scaling factor and a second offset, wherein the second scaling factor and the second offset are the scaling factor and the offset in the feature space transformation relationship from the visible light image to the infrared image.

[0282] In some embodiments of this disclosure, the step of aligning the feature maps of the visible light image and the infrared image in feature space may further include: performing element-wise multiplication of the feature map of the infrared image with a first scaling factor, and then summing it element-wise with a first offset to obtain the feature map of the adaptively transformed infrared image; performing element-wise multiplication of the feature map of the visible light image with a second scaling factor, and then summing it element-wise with a second offset to obtain the feature map of the adaptively transformed visible light image; performing element-wise summation of the feature map of the visible light image and the feature map of the adaptively transformed infrared image, and then performing a convolution operation to obtain the transfer feature map of the visible light image; performing element-wise summation of the feature map of the infrared image and the feature map of the adaptively transformed visible light image, and then performing a convolution operation to obtain the transfer feature map of the infrared image.

[0283] In some embodiments of this disclosure, step 93, which involves multi-level feature fusion of the feature maps of the visible light image and the infrared image, may include: d utilizing the differences between the feature maps of the visible light image and the infrared image to enhance the relationship between them.

[0284] In some embodiments of this disclosure, the step of enhancing the relationship between the feature maps of a visible light image and an infrared image by utilizing the differences between their feature maps may include: determining a first differential feature based on the difference between the feature maps of the infrared image and the visible light image, wherein the feature map of the infrared image is the output of the infrared extraction sub-model of the previous stage sub-model, and the feature map of the visible light image is the output of the visible light extraction sub-model of the previous stage sub-model; determining a second differential feature based on the difference between their feature maps; calculating a feature map to assist the visible light image based on the first differential feature, and calculating a feature map to assist the infrared image based on the second differential feature; performing element-level feature summation on the feature map of the visible light image and the feature map to assist the visible light image to determine a visible light image fusion feature map; and performing element-level feature summation on the feature map of the infrared image and the feature map to assist the infrared image to determine an infrared image fusion feature map.

[0285] In step 94, bounding box regression and classification are performed on the fused feature map to perform object detection.

[0286] In some embodiments of this disclosure, the target detection model includes a first network model, which includes multiple stage sub-models connected in series. Each stage sub-model is used to extract features from the visible light image and infrared image of the target, respectively, to obtain feature maps of the visible light image and the infrared image. The target detection method may further include: using the infrared light feature map output by a stage sub-model of the first network model as input to determine whether the current image was taken during the day or at night, and using internal feature representation to adjust the feature map output by each stage sub-model in the first network model.

[0287] In some embodiments of this disclosure, the step of using the infrared light feature map output by a stage sub-model of the first network model as input to determine whether the current image was taken during the day or at night, and using the internal feature representation to adjust the feature map output by each stage sub-model in the first network model, may include: performing average pooling on the feature map of the visible light image output by the first stage sub-model, then inputting it into two fully connected layers, and generating an internal feature representation through a first activation function; generating a first adjustment parameter and a second adjustment parameter for each stage sub-model based on the internal feature representation; and adjusting the feature map of each stage sub-model based on the first adjustment parameter and the second adjustment parameter of each stage sub-model.

[0288] In some embodiments of this disclosure, the target detection method may include: processing pre-processed RGB and infrared images using a trained multispectral fusion network to obtain high-precision pedestrian detection results; and processing visible light and infrared images at the current time using a target detection model to obtain target detection results, wherein the target detection model is trained according to the target detection model training method described in any of the above embodiments.

[0289] To address the poor performance of pedestrian detection under low light and nighttime conditions, this disclosure provides a pedestrian detection network (target detection model) based on illumination modality-driven multispectral differential information fusion, such as... Figure 2As shown, the fusion network includes a Backbone network, an auxiliary classification network (embedded with TCIM), a Neck network, and a Head network. In the Backbone network, to fully integrate features from different modalities, four fusion steps are performed, each using a cascaded structure of MAM and DMFM. MAM is a learning-based module that adaptively aligns the feature space. Then, DMFM utilizes the differences in modal information to enhance the relationships between different modalities, suppress common modalities, and amplify differential modalities. The auxiliary classification network uses the shallow feature map of the Backbone as input, and after passing through TCIM, determines whether the current image was taken during the day or at night, and then uses its internal feature representation to adjust the Backbone's feature map. Finally, the Backbone result is input into the Neck and Head to obtain the final pedestrian detection result. This network performs exceptionally well for both daytime and nighttime scenes, achieving a speed of up to 60 PFS.

[0290] This disclosure presents a pedestrian detection method based on illumination modality-driven multispectral differential information fusion. The method employs a novel network architecture, utilizing four fusion steps in the backbone to enhance the fusion of visible light and infrared images. It proposes a Modality Alignment Module (MAM) to align the feature spaces of different modalities and a Differential Modality Fusion Module (DMFM) to leverage the differences between modal information to improve feature fusion. Finally, a Task-Conditioned Illumination Module (TCIM) driven by day / night classification implicitly adjusts the network weights for multispectral pedestrian detection. In summary, the proposed method significantly improves pedestrian detection performance, especially in challenging scenarios such as nighttime and low-light conditions. The effectiveness of the proposed method is validated on the publicly available KAIST dataset with a loss rate of only 8.5%, and its generalization ability is validated on the publicly available CVC-14 dataset with a loss rate of only 18.3%. Furthermore, the method disclosed herein achieves speeds up to 60 FPS (frames per second), meeting real-time requirements and possessing significant engineering potential.

[0291] This disclosure utilizes both infrared and RGB images to achieve accurate pedestrian detection results under various harsh environmental conditions, such as at night and in low light, providing a reliable solution for fields such as nighttime surveillance and autonomous driving.

[0292] This disclosure proposes a pedestrian detection method based on illumination modality-driven multispectral differential information fusion. It cleverly designs a modality alignment module to align the feature spaces of different modalities, a differential information fusion module to enhance the fusion degree of different modalities, and a task-conditional illumination module to adjust the feature map of the multispectral pedestrian detection network according to the illumination intensity. This network can significantly improve pedestrian detection accuracy, with a loss rate of only 8.5% and 18.3% on the KAIST and CVC-14 datasets, respectively, and an inference speed of up to 60 FPS, showing great potential for mobile and embedded applications.

[0293] This disclosure proposes a novel object detection method (pedestrian detection method) based on illumination modality-driven multispectral differential information fusion. It comprehensively considers all the challenges encountered in the fusion process of visible light and infrared images, including feature extraction, feature space alignment, feature fusion, and weight adjustment of feature maps to distinguish between day and night scenes. Furthermore, this disclosure also proposes a multi-scale neck and a head for bounding box regression and classification.

[0294] Figure 10 These are schematic diagrams illustrating some embodiments of the object detection model construction apparatus of this disclosure. For example... Figure 10 As shown, the target detection model construction device of this disclosure may include a first construction module 101, a second construction module 102 and a third construction module 103.

[0295] The first construction module 101 is configured to construct a first network model, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0296] The second construction module 102 is configured to construct a second network model, wherein the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps.

[0297] The third construction module 103 is configured to construct a third network model, wherein the third network model is used to perform bounding box regression and classification, perform object detection, and predict the image features of the object. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to implement object detection.

[0298] In some embodiments of this disclosure, such as Figure 11 As shown, the target detection model training device of this disclosure may include a fourth building module 104.

[0299] The fourth construction module 104 is configured to construct a fourth network model, wherein the fourth network model is used to use the feature map of the first stage of the first network model as input to determine whether the current image was taken during the day or at night, and to adjust the feature map of the first network model using internal feature representation.

[0300] In some embodiments of this disclosure, the target detection model construction apparatus of this disclosure can be used to execute the target detection model construction method as described in any of the above embodiments of this disclosure.

[0301] According to another aspect of this disclosure, a target detection model training apparatus is provided, wherein the target detection model training apparatus is configured to train a target detection model using image training data, such that the trained target detection model is used to perform target detection, wherein the target detection model includes a first network model, a second network model, and a third network model.

[0302] Figure 11 These are schematic diagrams illustrating some embodiments of the object detection model training apparatus of this disclosure. For example... Figure 11 As shown, the target detection model training device of this disclosure may include a first training module 201, a second training module 202 and a third training module 203.

[0303] The first training module 201 is configured to perform feature extraction and fusion training on the first network model, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map.

[0304] The second training module 202 is configured to train a second network model using multi-scale feature fusion, wherein the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps.

[0305] The third training module 203 is configured to train a third network model for object detection. The third network model is used to perform bounding box regression and classification, perform object detection, and predict the image features of the object. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to achieve object detection.

[0306] In some embodiments of this disclosure, such as Figure 11 As shown, the target detection model training device of this disclosure may include a fourth training module 204.

[0307] The fourth training module 204 is configured to perform auxiliary classification training on the fourth network model, wherein the fourth network model is used to determine whether the current image was taken during the day or at night by using the feature map of the first stage of the first network model as input, and to adjust the feature map of the first network model by using internal feature representation.

[0308] In some embodiments of this disclosure, the target detection model training apparatus of this disclosure can be used to perform the target detection model training method as described in any of the above embodiments of this disclosure.

[0309] Figure 12 These are schematic diagrams illustrating some embodiments of the target detection device disclosed herein. For example... Figure 12 As shown, the target detection device of this disclosure may include an image receiving module 301 and a target detection module 302.

[0310] The image receiving module 301 is configured to receive visible light images and infrared images at the current moment.

[0311] The target detection module 302 is configured to extract features from the visible light image and the infrared image at the current time to obtain feature maps of the visible light image and the infrared image respectively; perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map; and perform bounding box regression and classification on the fused feature map to perform target detection.

[0312] In some embodiments of this disclosure, such as Figure 12 As shown, the target detection model training device of this disclosure may include a preprocessing module 303.

[0313] The preprocessing module 303 is configured to preprocess the visible light image and infrared image at the current moment.

[0314] In some embodiments of this disclosure, the target detection apparatus of this disclosure can be used to perform the target detection method as described in any of the above embodiments.

[0315] This disclosure also provides several device implementation schemes to complement the designed target detection method (pedestrian detection method), enabling more convenient application of the method to various fields. The implementation of this device gives the pedestrian detection method broader application potential, allowing it to demonstrate superior performance in various application scenarios and providing users in different fields with a convenient and reliable pedestrian detection tool.

[0316] Figure 13 This is a schematic diagram illustrating the structure of some embodiments of the computer device disclosed herein. For example... Figure 13 As shown, the computer device includes a memory 401 and a processor 402.

[0317] The memory 401 is used to store instructions, and the processor 402 is coupled to the memory 401. The processor 402 is configured to execute the object detection model construction method, the object detection model training method, or the object detection method as described in any of the above embodiments based on the instructions stored in the memory.

[0318] like Figure 13 As shown, the computer device also includes a communication interface 403 for exchanging information with other devices. Additionally, the computer device includes a bus 404, through which the processor 402, communication interface 403, and memory 401 communicate with each other.

[0319] The memory 401 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The memory 401 may also be a memory array. The memory 401 may also be divided into blocks, and these blocks may be combined into virtual volumes according to certain rules.

[0320] Furthermore, processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0321] This disclosure also provides a computer device including a processor and a memory. The processor performs processing tasks for visible light and infrared images and utilizes a convolutional neural network for image encoding and analysis. The memory stores image data and model parameters. In autonomous driving scenarios, this computer device can be used for pedestrian detection. The processor receives visible light and infrared image data and passes it to a multispectral fusion pedestrian detection network. This network employs a two-stream feature extractor and performs feature space alignment and fusion of different modalities during feature extraction. Then, the feature map is dynamically adjusted using illumination intensity to enhance the model's detection capabilities during both day and night. Finally, multi-scale features are fused through the Neck, and the bounding box location and category of pedestrian detection are generated through the Head. In addition to the processor and memory, the computer device can also integrate other components, such as a display, input devices, and network interfaces, to provide more comprehensive functionality and a richer user interaction experience. Users can input commands or image data through input devices, and the computer device processes the data using the multispectral fusion pedestrian detection network and displays the pedestrian detection results on the display or transmits them to other devices through the network interface. By providing this computer device, users can easily utilize pedestrian detection model results to provide accurate and real-time pedestrian detection and recognition capabilities for autonomous driving systems. This computer device has broad application prospects in fields such as autonomous vehicles and intelligent transportation systems.

[0322] Figure 14 These are schematic diagrams illustrating some embodiments of the target detection system disclosed herein. For example... Figure 14 As shown, the target detection device of this disclosure may include a visible light camera 501, an infrared camera 502, and a target detection device 503.

[0323] Visible light camera 501 is configured to acquire visible light images at the current moment.

[0324] Infrared camera 502 is configured to acquire infrared images of the current moment.

[0325] The target detection device 503 is the target detection device as described in any of the above embodiments.

[0326] In some embodiments of this disclosure, the target may be a pedestrian.

[0327] In some embodiments of this disclosure, the visible light camera and the infrared camera may be mounted on the top of the vehicle.

[0328] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the object detection model construction method, the object detection model training method, or the object detection method as described in any of the above embodiments.

[0329] In some embodiments of this disclosure, the computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0330] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the object detection model construction method, the object detection model training method, or the object detection method as described in any of the above embodiments.

[0331] The methods provided in this disclosure can also be configured as a program product including a readable medium storing the program code. This readable medium can take many forms, such as a readable signal medium or a readable storage medium, such as an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. Specifically, the readable storage medium can include, but is not limited to, electrical connections, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0332] The target detection model building apparatus, first building module, second building module, third building module, fourth building module, target detection model training apparatus, first training module, second training module, third training module, fourth training module, target detection apparatus, image acquisition module, target detection module, and computer device described above can be implemented as a general-purpose processor, programmable logic controller (PLC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in this application.

[0333] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0334] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0335] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for constructing an object detection model, comprising: A first network model is constructed, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and multi-level feature fusion is performed on the feature maps of the visible light image and the infrared image to obtain a fused feature map. A second network model is constructed, wherein the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps; A third network model is constructed, wherein the third network model is used to perform bounding box regression and classification for object detection. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to achieve object detection.

2. The target detection model construction method according to claim 1, wherein, The first network model includes multiple stage sub-models connected in series, and each stage sub-model includes: The visible light extraction sub-model is used to extract features from the visible light image of the target or the feature map of the visible light image output by the previous stage sub-model, so as to obtain the feature map of the visible light image of the current stage sub-model. The infrared extraction sub-model is used to extract features from the infrared image of the target or the feature map of the infrared image of the previous stage sub-model to obtain the feature map of the infrared image of the current stage sub-model. The fusion sub-model is used to fuse the feature maps of the visible light image and the infrared image of the current stage sub-model.

3. The target detection model construction method according to claim 2, wherein, The construction of the first network model includes: Multiple stage sub-models are constructed in series. The first network model includes n stage sub-models, where n is a natural number greater than 1. Each stage sub-model includes a visible light extraction sub-model and an infrared extraction sub-model. The visible light extraction sub-model and infrared extraction sub-model of the first stage sub-model are used to perform convolution operations and downsampling on the input visible light image and infrared image to generate the feature map of the first stage. The visible light extraction sub-model and infrared extraction sub-model of the i-th stage sub-model are used to perform convolution operations and downsampling on the feature map generated by the (i-1)-th stage sub-model to generate the feature map of the i-th stage, where i is a natural number greater than 1 and less than or equal to n.

4. The target detection model construction method according to claim 3, wherein, The construction of the series of multiple stage sub-models includes: Multiple modality alignment modules are constructed, wherein the fusion sub-model includes a modality alignment module, which is connected to the visible light extraction sub-model and the infrared extraction sub-model respectively. The modality alignment module is used to perform feature space alignment on the feature maps of the visible light image and the infrared image.

5. The target detection model construction method according to claim 4, wherein, The modality alignment module includes: The feature concatenation model is used to concatenate feature maps of visible light images and infrared images to generate concatenated feature maps. A first transformation network is used to process the cascaded feature map to obtain a first scaling factor and a first offset, wherein the first scaling factor and the first offset are the scaling factor and offset in the feature space transformation relationship from infrared image to visible light image; The second transformation network is used to process the cascaded feature map to obtain a second scaling factor and a second offset, wherein the second scaling factor and the second offset are the scaling factor and offset in the feature space transformation relationship from visible light image to infrared image.

6. The target detection model construction method according to claim 5, wherein, The modality alignment module also includes: The feature space adaptation operation model is used to perform element-wise multiplication of the feature map of the infrared image with the first scaling factor, and then sum the feature map of the infrared image with the first offset to obtain the feature map of the infrared image after adaptive transformation; and to perform element-wise multiplication of the feature map of the visible light image with the second scaling factor, and then sum the feature map of the visible light image with the second offset to obtain the feature map of the visible light image after adaptive transformation. The convolution model is used to sum the feature maps of the visible light image and the adaptively transformed infrared image element-wise, and then perform a convolution operation to obtain the transfer feature map of the visible light image. Similarly, the feature maps of the infrared image and the adaptively transformed visible light image are summed element-wise and then convolved to obtain the transfer feature map of the infrared image.

7. The method for constructing a target detection model according to any one of claims 2 to 6, wherein, The construction of the series of interconnected stage sub-models includes: Multiple differential information fusion modules are constructed, wherein the fusion sub-model includes a differential information fusion module, which is connected to the visible light extraction sub-model and the infrared extraction sub-model respectively. The differential information fusion module is used to enhance the relationship between the feature maps of the visible light image and the infrared image by utilizing the differences between the feature maps of the visible light image and the infrared image.

8. The target detection model construction method according to claim 7, wherein, The differential information fusion module includes: A differential feature determination model is used to determine the first differential feature based on the difference between the feature map of the infrared image and the feature map of the visible light image, wherein the feature map of the infrared image is the output of the infrared extraction sub-model of the previous stage sub-model, and the feature map of the visible light image is the output of the visible light extraction sub-model of the previous stage sub-model; and to determine the second differential feature based on the difference between the feature maps of the visible light image and the infrared image. An auxiliary feature determination model is used to calculate a feature map to assist the visible light image based on the first difference feature, and to calculate a feature map to assist the infrared image based on the second difference feature. The differential fusion model is used to sum the element-wise features of the feature map of the visible light image and the feature map used to assist the visible light image to determine the fused feature map of the visible light image, and to sum the element-wise features of the feature map of the infrared image and the feature map used to assist the infrared image to determine the fused feature map of the infrared image.

9. The target detection model construction method according to claim 8, wherein, The auxiliary feature determination model includes: The weight factor generation module is used to perform global average pooling on the first difference feature, scale and restore the channels through convolution, and generate the first weight factor through an activation function; and to perform global average pooling on the second difference feature, scale and restore the channels through convolution, and generate the second weight factor through an activation function. The auxiliary feature determination module is used to perform element-level feature integration between the first weight factor and the feature map of the infrared image to obtain a feature map used to assist the visible light image; and to perform element-level feature integration between the second weight factor and the feature map of the visible light image to obtain a feature map used to assist the infrared image.

10. The method for constructing a target detection model according to any one of claims 3 to 6, wherein, The construction of the series of interconnected stage sub-models includes: A stitching module is constructed, wherein the fusion sub-model of the nth stage sub-model includes the stitching module, and the stitching module is used to stitch together the feature map of the visible light image and the feature map of the infrared image generated by the (n-1)th stage sub-model.

11. The target detection model construction method according to claim 7, wherein, The fusion sub-model includes a modality alignment module and a differential information fusion module, and the construction of multiple cascaded sub-models includes: An input determination module is constructed, wherein the input determination module is used to determine that the input of the visible light extraction sub-model in the i-th stage sub-model is the sum of the original input of the visible light extraction sub-model, the transfer feature map of the visible light image after passing through the modal alignment module of the (i-1)-th stage sub-model, and the fusion feature map of the visible light image after passing through the differential information fusion module of the (i-1)-th stage sub-model; and to determine that the input of the infrared extraction sub-model in the i-th stage sub-model is the sum of the original input of the infrared extraction sub-model, the transfer feature map of the infrared image after passing through the modal alignment module of the (i-1)-th stage sub-model, and the fusion feature map of the infrared image after passing through the differential information fusion module of the (i-1)-th stage sub-model, wherein the first network model includes n stage sub-models, where n is a natural number greater than 1, and i is a natural number greater than 1 and less than or equal to n.

12. The target detection model construction method according to any one of claims 2 to 6, further comprising: A fourth network model is constructed, wherein the fourth network model is connected to the first network model, and is used to determine whether the current image was taken during the day or at night by using the feature map output by the visible light extraction sub-model of the first network model as input, and to adjust the feature map output by each stage sub-model in the first network model by using internal feature representation.

13. The target detection model construction method according to claim 12, wherein, The construction of the fourth network model includes: A feature representation generation module is constructed, wherein the feature representation generation module is used to perform average pooling on the feature map of the visible light image of the visible light extraction sub-model in the first stage sub-model, and then input it into two fully connected layers, and generate internal feature representation through the first activation function; An adjustment parameter generation module is constructed, wherein the adjustment parameter generation module is used to generate the first adjustment parameter and the second adjustment parameter of each stage sub-model based on the internal feature representation; A feature map adjustment module is constructed, wherein the feature map adjustment module is used to adjust the feature map of each stage sub-model according to the first adjustment parameter and the second adjustment parameter of each stage sub-model.

14. The target detection model construction method according to claim 13, wherein, The adjustment parameter generation module includes: The first adjustment parameter generation submodule is used to determine the weights and biases corresponding to the first adjustment parameters of the k-th stage sub-model; and to determine the first adjustment parameters based on the weights and biases corresponding to the first adjustment parameters, the internal feature representation, and the first activation function. The second adjustment parameter generation submodule is used to determine the weights and biases corresponding to the second adjustment parameters of the k-th stage sub-model; and to determine the second adjustment parameters based on the weights and biases corresponding to the second adjustment parameters, the internal feature representation, and the first activation function.

15. The target detection model construction method according to claim 14, wherein, The feature map adjustment module includes: The third adjustment parameter generation submodule is used to determine the third adjustment parameter based on the difference between 1 and the first adjustment parameter; The feature map adjustment submodule is used to multiply the third adjustment parameter with the feature map of the k-th stage sub-model, then sum it with the second adjustment parameter, and input it into the first activation function to obtain the adjusted feature map of the k-th stage sub-model.

16. The method for constructing a target detection model according to any one of claims 2 to 6, wherein, The construction of the second network model includes: A top-down structure is constructed, wherein the top-down structure is used to upsample the feature map output by the first network model, and the upsampled feature map is merged with the feature maps output by the visible light extraction sub-model and the infrared extraction sub-model in the first network model after feature extraction; A bottom-up structure is constructed, wherein the bottom-up structure is used to stitch together multi-layer feature maps, and the multi-layer feature maps are feature maps that have been processed by the top-down structure at different times.

17. The method for constructing a target detection model according to any one of claims 1 to 6, wherein, The construction of the third network model includes: Multiple detection heads of different sizes are constructed to cover targets of different scales. Each detection head is used to predict whether a target exists within the grid and to determine target information, including the target's location, size, and category.

18. A method for training an object detection model, comprising: The target detection model is trained using image training data, so that the trained target detection model can be used to perform target detection. The target detection model includes a first network model, a second network model, and a third network model. The step of training the target detection model using image data includes: The first network model is subjected to feature extraction and fusion training. The first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image. Multi-level feature fusion is performed on the feature maps of the visible light image and the infrared image to obtain a fused feature map. The second network model is trained to perform multi-scale feature fusion, wherein the second network model is used to obtain multi-scale feature maps of the fused feature map and to perform multi-scale feature fusion on the multi-scale feature maps; The third network model is trained for object detection. The third network model is used to perform bounding box regression and classification to perform object detection. The object detection model includes the first network model, the second network model and the third network model. The trained object detection model is used to achieve object detection.

19. The target detection model training method according to claim 18, wherein, The target detection model is constructed using the target detection model construction method as described in any one of claims 1-17.

20. A target detection method, wherein the target detection method is executed by a target detection model, wherein, The target detection method includes: Receive the visible light and infrared images at the current moment; Feature extraction is performed on the visible light image and infrared image at the current moment to obtain the feature map of the visible light image and the feature map of the infrared image respectively; Multi-level feature fusion is performed on the feature maps of visible light images and infrared images to obtain a fused feature map. Object detection is performed by performing bounding box regression and classification on the fused feature maps.

21. The target detection method according to claim 20, wherein, The multi-level feature fusion of the feature maps of visible light images and infrared images includes: Feature space alignment is performed on the feature maps of visible light images and infrared images.

22. The target detection method according to claim 21, wherein, The feature space alignment of the feature maps of the visible light image and the infrared image includes: The feature maps of visible light images and infrared images are concatenated to generate a concatenated feature map. The cascaded feature map is processed to obtain a first scaling factor and a first offset, wherein the first scaling factor and the first offset are the scaling factor and offset in the feature space transformation relationship from infrared image to visible light image; The cascaded feature map is processed to obtain a second scaling factor and a second offset, wherein the second scaling factor and the second offset are the scaling factor and offset in the feature space transformation relationship from visible light image to infrared image.

23. The target detection method according to claim 22, wherein, The feature space alignment of the feature maps of the visible light image and the infrared image further includes: After performing element-wise multiplication of the feature map of the infrared image with the first scaling factor, and then summing it element-wise with the first offset, the feature map of the infrared image after adaptive transformation is obtained. After performing element-wise multiplication of the feature map of the visible light image with the second scaling factor, and then summing it element-wise with the second offset, the feature map of the visible light image after adaptive transformation is obtained. The feature map of the visible light image and the feature map of the adaptively transformed infrared image are summed element-wise and then convolved to obtain the transfer feature map of the visible light image. The feature map of the infrared image is obtained by summing the feature map of the adaptively transformed visible light image element-wise and then performing a convolution operation.

24. The target detection method according to any one of claims 20 to 23, wherein, The multi-level feature fusion of the feature maps of visible light images and infrared images includes: The relationship between the feature maps of visible light images and infrared images can be enhanced by utilizing the differences between their feature maps.

25. The target detection method according to claim 24, wherein, The method of enhancing the relationship between the feature maps of visible light images and infrared images by utilizing the differences between their feature maps includes: The first differential feature is determined based on the difference between the feature maps of the infrared image and the visible light image, where the feature map of the infrared image is the output of the infrared extraction sub-model of the previous stage sub-model, and the feature map of the visible light image is the output of the visible light extraction sub-model of the previous stage sub-model; the second differential feature is determined based on the difference between the feature maps of the visible light image and the infrared image. The feature map used to assist the visible light image is calculated based on the first difference feature, and the feature map used to assist the infrared image is calculated based on the second difference feature. The visible light image feature map is determined by summing the element-level features of the feature map used to assist the visible light image; the infrared image feature map is determined by summing the element-level features of the feature map used to assist the infrared image.

26. The target detection method according to any one of claims 20 to 23, wherein, The target detection model includes a first network model, which includes multiple stage sub-models connected in series. Each stage sub-model is used to extract features from the visible light image and infrared image of the target, respectively, to obtain feature maps of the visible light image and the infrared image. The target detection method further includes: Using the infrared light feature map output by a stage sub-model of the first network model as input, it is determined whether the current image was taken during the day or at night, and the feature map output by each stage sub-model in the first network model is adjusted using internal feature representation.

27. The target detection method according to claim 26, wherein, The step of using the infrared light feature map output by a stage sub-model of the first network model as input to determine whether the current image was taken during the day or at night, and adjusting the feature map output by each stage sub-model in the first network model using internal feature representation, includes: The feature map of the visible light image output by the first stage sub-model is average pooled and then input into two fully connected layers. The internal feature representation is generated by passing through the first activation function. Based on the internal feature representation, generate the first and second adjustment parameters for each stage sub-model; The feature map of each stage sub-model is adjusted according to the first and second adjustment parameters of each stage sub-model.

28. A target detection model construction apparatus, comprising: The first construction module is configured to construct a first network model, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map. The second construction module is configured to construct a second network model, wherein the second network model is used to obtain multi-scale feature maps of the fused feature maps and to perform multi-scale feature fusion on the multi-scale feature maps; The third building module is configured to build a third network model, wherein the third network model is used to perform bounding box regression and classification for object detection. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to implement object detection.

29. A target detection model training device, wherein, The target detection model training device is configured to train the target detection model using image training data, so that the trained target detection model can be used to perform target detection. The target detection model includes a first network model, a second network model, and a third network model. The target detection model training device includes: The first training module is configured to perform feature extraction and fusion training on the first network model, wherein the first network model is used to extract features from the visible light image and the infrared image of the target respectively to obtain feature maps of the visible light image and the infrared image; and to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map. The second training module is configured to train the second network model to perform multi-scale feature fusion, wherein the second network model is used to obtain the multi-scale feature map of the fused feature map and perform multi-scale feature fusion on the multi-scale feature map; The third training module is configured to train a third network model for object detection. The third network model is used to perform bounding box regression and classification to perform object detection. The object detection model includes a first network model, a second network model, and a third network model. The trained object detection model is used to implement object detection.

30. A target detection device, comprising: The image receiving module is configured to receive visible light and infrared images at the current moment; The target detection module is configured to extract features from the visible light image and the infrared image at the current time, respectively, to obtain feature maps of the visible light image and the infrared image; to perform multi-level feature fusion on the feature maps of the visible light image and the infrared image to obtain a fused feature map; and to perform bounding box regression and classification on the fused feature map to perform target detection.

31. A computer device comprising: The memory is configured to store instructions; A processor is configured to execute the instructions such that the computer device implements the object detection model construction method as described in any one of claims 1-17, the object detection model training method as described in claim 18 or 19, or the object detection method as described in any one of claims 20-27.

32. A target detection system, comprising: A visible light camera is configured to capture a visible light image of the current moment. An infrared camera is configured to capture infrared images of the current moment. The target detection device according to claim 30.

33. The target detection system according to claim 32, wherein: The target is pedestrians; And / or, The visible light camera and the infrared camera are mounted on the top of the vehicle.

34. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the object detection model construction method as described in any one of claims 1-17, the object detection model training method as described in claim 18 or 19, or the object detection method as described in any one of claims 20-27.

35. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the target detection model construction method as described in any one of claims 1-16, the target detection model training method as described in claim 18 or 19, or the target detection method as described in any one of claims 20-27.

Images