Infrared and visible adaptive target detection method based on selective expert routing

Abstract

The application relates to an infrared and visible light adaptive target detection method based on selective expert routing. The method comprises the following steps: acquiring infrared images and visible light images, and performing different-scale shallow layer coding on the infrared images and the visible light images to obtain first initial multi-modal features and second initial multi-modal features; determining a modal confidence vector according to the infrared images and the visible light images, and determining at least one target expert branch; inputting the first initial multi-modal features and the second initial multi-modal features into the at least one target expert branch to obtain at least one detection feature; based on a cross-modal cross-attention mechanism, performing region-level feature interaction on each detection feature to obtain a candidate box of each target expert branch, and weighting and integrating a classification probability and a regression result of the candidate box of each target expert branch to obtain a target detection result. Thus, the problems of poor modal adaptability, fusion redundancy, waste of computing resources, insufficient detection robustness and insufficient reasoning efficiency in the related art are solved, and efficient and high-quality target detection is realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an infrared and visible light adaptive target detection method based on selective expert routing. Background Technology

[0002] With the widespread adoption of multimodal sensors, infrared and visible light target detection have become key technologies in complex environments. Infrared imaging can highlight the thermal features of targets in low light, nighttime, and occluded conditions, while visible light imaging has a natural advantage in depicting details such as texture and color. Existing research generally involves directly fusing the features of the two modalities, hoping to obtain a stronger complementary effect.

[0003] However, the following significant problems exist in the related technologies: (1) Modal rigidity: Most multimodal target detection methods require the input to contain both infrared and visible light images simultaneously, otherwise the model cannot function properly. This severely limits its robustness and flexibility in practical applications. (2) Fusion redundancy: Although fusion can theoretically utilize complementary information, it may introduce interference in practical scenarios. For example, infrared features have no significant gain under strong sunlight during the day, and may even lead to redundancy or noise in the fused features; while in rainy or foggy weather, the detection performance may be reduced due to the large difference in signal-to-noise ratio between modes. (3) Waste of computational resources: Traditional multimodal fusion networks often perform full feature extraction and fusion of all modes in each inference process, resulting in unnecessary computational overhead and making it difficult to meet the application requirements of edge detection, real-time detection, etc. Therefore, how to achieve adaptive mode selection, that is, intelligently determine whether to use a single mode or a fused mode in the current scenario, and significantly improve efficiency while ensuring detection accuracy, is a core problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides an infrared and visible light adaptive target detection method based on selective expert routing to solve the problems of poor modal adaptability, fusion redundancy, waste of computational resources, and insufficient detection robustness and inference efficiency in related technologies, so as to achieve efficient and high-quality target detection.

[0005] To achieve the above objectives, the first aspect of this application proposes an infrared and visible light adaptive target detection method based on selective expert routing, comprising the following steps:

[0006] Infrared and visible light images are acquired, and heteroscale shallow coding is performed on the infrared and visible light images respectively to obtain a first initial multimodal feature and a second initial multimodal feature; A modal confidence vector is determined based on the infrared image and the visible light image, and at least one target expert branch is determined based on the modal confidence vector. The first initial multimodal feature and the second initial multimodal feature are input into the at least one target expert branch to obtain at least one detection feature. Based on a pre-defined cross-modal attention mechanism, regional feature interaction is performed on each detection feature to obtain candidate boxes for each target expert branch. The target detection result is obtained by weighted integration based on the classification probability and regression results of the candidate boxes of each target expert branch.

[0007] According to one embodiment of this application, determining the modality confidence vector based on the infrared image and the visible light image includes: Modal availability discrimination and illumination quality assessment are performed on the infrared image and the visible light image respectively to obtain modal availability discrimination results and illumination quality assessment results; Based on the modal availability discrimination results and illumination quality assessment results, the conditional vector of the infrared image and the conditional vector of the visible light image are determined. A modal confidence vector is generated based on the conditional vector of the infrared image and the conditional vector of the visible light image.

[0008] According to one embodiment of this application, determining at least one target expert branch based on the modality confidence vector includes: A database of expert branches is defined, which includes visible light experts, infrared experts, and fusion experts. The modal confidence vector, the infrared image, and the visible light image are input to a preset routing controller, so that the preset routing controller outputs the activation weights of the visible light expert, the infrared expert, and the fusion expert, respectively. The at least one target expert branch is determined based on the activation weights of the visible light expert, the infrared expert, and the fusion expert.

[0009] According to one embodiment of this application, the weighted ensemble result of the classification probabilities of the candidate boxes is: ; in, P The weighted ensemble result of the classification probabilities of the candidate boxes. For the activated expert branch, For the first e Routing weights for each expert branch For the first e The classification probabilities of candidate boxes output by each expert branch.

[0010] According to one embodiment of this application, the weighted integration result of the regression results of the candidate boxes is as follows: ; in, The weighted integration result of the regression results of the candidate boxes. The candidate box regression result is output by the e-th expert branch.

[0011] The infrared and visible light adaptive target detection method based on selective expert routing proposed in this application involves performing heteroscale shallow encoding on infrared and visible light images to obtain first and second initial multimodal features, determining modality confidence vectors, and determining target expert branches based on these vectors. The first and second initial multimodal features are then input into the target expert branches to obtain detection features. Based on a cross-modal cross-attention mechanism, region-level feature interaction is performed on the detection features to obtain candidate boxes. Finally, the target detection result is obtained by weighted ensemble based on the classification probabilities and regression results of the candidate boxes. This method solves the problems of poor modality adaptability, fusion redundancy, wasted computational resources, and insufficient detection robustness and inference efficiency in related technologies, achieving efficient and high-quality target detection.

[0012] To achieve the above objectives, a second aspect of this application provides an infrared and visible light adaptive target detection device based on selective expert routing, comprising: The acquisition module acquires infrared images and visible light images, and performs heteroscale shallow coding on the infrared images and the visible light images respectively to obtain first initial multimodal features and second initial multimodal features; The determination module determines a modal confidence vector based on the infrared image and the visible light image, and determines at least one target expert branch based on the modal confidence vector. The first initial multimodal feature and the second initial multimodal feature are input into the at least one target expert branch to obtain at least one detection feature. The detection module, based on a preset cross-modal cross-attention mechanism, performs regional-level feature interaction on each detection feature to obtain candidate boxes for each target expert branch, and then weights and integrates the candidate boxes of each target expert branch according to their classification probability and regression results to obtain the target detection result.

[0013] According to one embodiment of this application, the determining module is specifically used for: Modal availability discrimination and illumination quality assessment are performed on the infrared image and the visible light image respectively to obtain modal availability discrimination results and illumination quality assessment results; Based on the modal availability discrimination results and illumination quality assessment results, the conditional vector of the infrared image and the conditional vector of the visible light image are determined. A modal confidence vector is generated based on the conditional vector of the infrared image and the conditional vector of the visible light image.

[0014] According to one embodiment of this application, the determining module is specifically used for: A database of expert branches is defined, which includes visible light experts, infrared experts, and fusion experts. The modal confidence vector, the infrared image, and the visible light image are input to a preset routing controller, so that the preset routing controller outputs the activation weights of the visible light expert, the infrared expert, and the fusion expert, respectively. The at least one target expert branch is determined based on the activation weights of the visible light expert, the infrared expert, and the fusion expert.

[0015] According to one embodiment of this application, the weighted ensemble result of the classification probabilities of the candidate boxes is: ; in, P The weighted ensemble result of the classification probabilities of the candidate boxes. For the activated expert branch, For the first e Routing weights for each expert branch For the first e The classification probabilities of candidate boxes output by each expert branch.

[0016] According to one embodiment of this application, the weighted integration result of the regression results of the candidate boxes is as follows: ; in, The weighted integration result of the regression results of the candidate boxes. For the first e The candidate box regression results output by each expert branch.

[0017] The infrared and visible light adaptive target detection device based on selective expert routing proposed in this application performs heteroscale shallow encoding on infrared and visible light images to obtain first and second initial multimodal features, determines modality confidence vectors, and determines target expert branches based on the modality confidence vectors. The first and second initial multimodal features are input into the target expert branches to obtain detection features. Based on a cross-modal cross-attention mechanism, the detection features are subjected to region-level feature interaction to obtain candidate boxes. The target detection result is obtained by weighted integration based on the classification probability and regression results of the candidate boxes. This solves the problems of poor modality adaptability, fusion redundancy, waste of computational resources, and insufficient detection robustness and inference efficiency in related technologies, achieving efficient and high-quality target detection.

[0018] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the infrared and visible light adaptive target detection method based on selective expert routing as described in the above embodiments.

[0019] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the infrared and visible light adaptive target detection method based on selective expert routing as described in the above embodiments.

[0020] To achieve the above objectives, a fifth aspect of this application provides a computer program product, which, when executed by a processor, implements the infrared and visible light adaptive target detection method based on selective expert routing as described in the above embodiments.

[0021] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an infrared and visible light adaptive target detection method based on selective expert routing provided in an embodiment of this application; Figure 2 This is a flowchart of an infrared and visible light adaptive target detection method based on selective expert routing according to an embodiment of this application; Figure 3 This is a block diagram of an infrared and visible light adaptive target detection device based on selective expert routing according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0024] The infrared and visible light adaptive target detection method based on selective expert routing proposed according to the embodiments of this application will be described below with reference to the accompanying drawings. First, the infrared and visible light adaptive target detection method based on selective expert routing proposed according to the embodiments of this application will be described with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart of an embodiment of the infrared and visible light adaptive target detection method based on selective expert routing according to this application.

[0026] like Figure 1 As shown, the infrared and visible light adaptive target detection method based on selective expert routing includes the following steps: In step S101, infrared images and visible light images are acquired, and heteroscale shallow coding is performed on the infrared images and visible light images respectively to obtain the first initial multimodal features and the second initial multimodal features.

[0027] Heterogeneous-scale shallow coding refers to the process of independently extracting and encoding features at different scales or adaptive scales in the shallow stages of a neural network for images of different modalities. The first initial multimodal feature refers to the initial feature representation extracted from the infrared image through heterogeneous-scale shallow coding. The second initial multimodal feature refers to the initial feature representation extracted from the visible light image through heterogeneous-scale shallow coding.

[0028] Specifically, embodiments of this application acquire low-resolution infrared images ( ) and high-resolution visible light images ( )(in s (For the super-resolution scaling factor), two independent shallow encoders are used. and Lightweight shallow coding is performed on infrared and visible light images to obtain the first initial multimodal features. Second initial multimodal features The shallow encoder consists of several convolutional layers and is used to capture low-level structural information such as edges and gradients. Its formula is: ; ; in, This is the first initial multimodal feature. It is an infrared shallow encoder. For low-resolution infrared images, For a three-dimensional real number feature space, This is the second initial multimodal feature. Encoding for the shallow layer of visible light. For high-resolution visible light images, It is a three-dimensional real number feature space.

[0029] In step S102, a modal confidence vector is determined based on the infrared image and the visible light image, and at least one target expert branch is determined based on the modal confidence vector. The first initial multimodal feature and the second initial multimodal feature are input to the at least one target expert branch to obtain at least one detection feature.

[0030] Here, the modality confidence vector refers to a multi-dimensional numerical vector obtained after quantitatively evaluating the effective information representation capability, feature reliability, and scene adaptability of the modal image. The target expert branch refers to a dedicated feature processing branch adapted to the current infrared-visible dual-modal scene characteristics. The detection feature refers to the deep feature representation output by the target expert branch after performing targeted processing, fusion, and optimization of the initial multimodal features of the infrared and visible dual-modal scenes.

[0031] Specifically, in this embodiment, based on the input infrared and visible light images, a modal confidence vector is obtained by quantitatively evaluating the usability and effectiveness of the dual-modal images. Then, based on the modal confidence vector, at least one target expert branch that is suitable for the feature processing requirements of the current dual-modal scene is selected. The first initial multimodal feature and the second initial multimodal feature obtained by shallow encoding in step S101 are input into the at least one target expert branch. After the target expert branch performs dedicated feature processing and fusion optimization, at least one detection feature that meets the requirements of the subsequent target detection task is generated.

[0032] Optionally, in some embodiments, determining the modal confidence vector based on the infrared image and the visible light image includes: performing modal availability discrimination and illumination quality assessment on the infrared image and the visible light image respectively to obtain modal availability discrimination results and illumination quality assessment results; determining the condition vector of the infrared image and the condition vector of the visible light image based on the modal availability discrimination results and the illumination quality assessment results; and generating the modal confidence vector based on the condition vector of the infrared image and the condition vector of the visible light image.

[0033] Modal availability discrimination refers to the quantitative analysis and discrimination process that determines whether infrared and visible light modal images can provide effective information support for subsequent multimodal feature processing, fusion, and target detection tasks.

[0034] Specifically, this application embodiment uses indicators such as brightness estimation, sharpness analysis, and noise level calculation to determine the usability of visible light and infrared images in the current scene. Based on the usability discrimination and illumination quality assessment of the input infrared and visible light image modes, it identifies the validity of the modes in the current scene and determines the condition vector of the infrared image. c ir Conditional vectors of visible light images cvis And output modal confidence vector. C = [ c vis , c ir ].

[0035] Optionally, in some embodiments, determining at least one target expert branch based on the modal confidence vector includes: determining an expert branch library, which includes visible light experts, infrared experts, and fusion experts; inputting the modal confidence vector, the infrared image, and the visible light image to a preset routing controller, so that the preset routing controller outputs the activation weights of the visible light experts, the infrared experts, and the fusion experts, respectively; and determining at least one target expert branch based on the activation weights of the visible light experts, the infrared experts, and the fusion experts.

[0036] Here, the preset routing controller refers to a dedicated intelligent control module that has been pre-designed and configured with algorithm logic and parameters. The activation weight refers to the specific quantized weight value output by the preset routing controller for each expert branch in the expert branch library.

[0037] Specifically, this application embodiment introduces a lightweight gating network as a routing controller to manage the modal confidence vector. C After concatenation with the global pooling result of shallow features, the result is input into a lightweight gating network. The lightweight gating network dynamically generates expert activation weight vectors based on modality confidence and content features of the infrared and visible light images. Through a dynamic selection strategy, this embodiment activates the corresponding visible light expert, infrared expert, or fusion expert only when necessary. For example, only the optimal 1-2 expert branches can be activated, avoiding unnecessary computational overhead and thus completing dynamic routing decisions.

[0038] The lightweight gating network is jointly trained with the object detection network through end-to-end training. During model training, the lightweight gating network and each expert branch network are trained through joint optimization, and their network parameters are updated using a backpropagation algorithm based on a pre-defined joint loss function. After model training, the lightweight gating network is embedded and applied to the inference phase to adaptively generate activation weights for expert branches based on the input infrared and visible light images, thereby achieving dynamic routing decisions. Furthermore, as one of the core architectures of the pre-defined routing controller, the gating network adopts a lightweight feedforward neural network structure, forming a clearly hierarchical and interconnected overall network system with the shallow feature encoding module, expert branch processing module, and detection output module. The gating network consists of an input layer, a feature mapping layer, a weight generation layer, and a normalization layer connected sequentially. The input layer receives modal confidence vectors obtained from infrared and visible light images through modal availability discrimination and illumination quality assessment, as well as content features obtained from shallow features through global pooling. The feature mapping layer performs nonlinear modeling and dimensionality compression on the gating input features. The weight generation layer generates expert activation scores that correspond one-to-one with the expert branch library. The normalization layer normalizes the activation scores and outputs expert activation weight vectors. These expert activation weights are applied to the expert branch processing module through differentiable connections, controlling the selection and participation levels of visible light experts, infrared experts, and fusion experts, thereby achieving dynamic expert routing based on scene and modal characteristics. Thus, through the above module structure, hierarchical design, and connection relationships, the gating network undertakes the core functions of expert selection and computational resource allocation in the overall model, defining the model's selective expert routing structure and adaptive detection mechanism.

[0039] Furthermore, in this embodiment, the first initial multimodal features and the second initial multimodal features are fed into at least one target expert branch for task specialization processing. Specifically, the visible light expert employs a multi-scale convolutional structure to enhance texture details and edge contours, suitable for high-brightness and high-resolution scenes; the infrared expert utilizes thermal pattern modeling and background suppression mechanisms to enhance the separability of targets in low-light and nighttime conditions; and the fusion expert is designed as a cross-modal interaction module, using a dual-stream attention mechanism to extract complementary information and align semantic features. Thus, each expert branch outputs a task-specialized detection feature map. .

[0040] In step S103, based on the preset cross-modal cross-attention mechanism, regional feature interaction is performed on each detection feature to obtain the candidate box of each target expert branch, and the target detection result is obtained by weighted integration based on the classification probability and regression result of the candidate box of each target expert branch.

[0041] Optionally, in some embodiments, the weighted ensemble result of the classification probabilities of the candidate boxes is: ; in, P This is the weighted ensemble result of the classification probabilities of the candidate boxes. For the activated expert branch, For the first e Routing weights for each expert branch For the first e The classification probabilities of candidate boxes output by each expert branch.

[0042] Optionally, in some embodiments, the weighted integration result of the regression results of the candidate boxes is: ; in, The weighted ensemble result of the regression results for the candidate boxes. For the first e The candidate box regression results output by each expert branch.

[0043] Among them, regional feature interaction refers to cross-modal feature association, fusion and enhancement operations carried out on the detection features of target detection with image spatial regions as the basic processing units.

[0044] Specifically, this application embodiment adopts a cross-modal cross-attention mechanism to perform regional-level feature interaction on the detection features of expert branches to obtain candidate boxes for each target expert branch. Furthermore, this application embodiment uses the center position of the candidate region as the query to perform weighted aggregation on features from different experts. Visible light and infrared features in salient regions receive higher fusion weights, while redundant features in background regions are suppressed, making the candidate boxes more discriminative and robust, and improving the accuracy of target localization in complex scenes.

[0045] Furthermore, based on the routing weights assigned by the expert routing module, the target category probabilities output by the multimodal expert branches are weighted and fused with the location regression parameters, i.e., the classification probabilities of the candidate boxes output by each expert branch are obtained independently. Regression results According to the routing weights in this application embodiment The detection results of all activated branches are weighted and integrated to obtain an adaptive target detection output. When some modal inputs are missing or the perception is unreliable due to environmental interference, the routing weights can adaptively adjust the participation level of each expert branch, so that the embodiments of this application can maintain high detection accuracy and stability without relying on a single modality, thereby achieving robust target detection for complex and incomplete input conditions.

[0046] Therefore, by performing heteroscale shallow encoding on infrared and visible light images respectively to obtain first and second initial multimodal features, and determining modality confidence vectors, a target expert branch is determined based on the modality confidence vectors. The first and second initial multimodal features are then input into the target expert branch to obtain detection features. Based on a cross-modal cross-attention mechanism, region-level feature interaction is performed on the detection features to obtain candidate boxes. Finally, the target detection result is obtained by weighted ensemble based on the classification probability and regression results of the candidate boxes. This solves the problems of poor modality adaptability, fusion redundancy, waste of computational resources, and insufficient detection robustness and inference efficiency in related technologies, achieving efficient and high-quality target detection.

[0047] Furthermore, in order to balance the accuracy, computational efficiency, and scene generalization ability of infrared and visible light modal target detection, this application adopts an end-to-end joint training framework and multi-loss joint optimization to enable the detection network to have scene-adaptive and task-driven detection performance.

[0048] This application also constructs an end-to-end joint training framework, which comprehensively utilizes detection loss, route sparsity regularization, and modality consistency constraints for joint optimization. Among these, the detection loss... Including classification loss Regression loss Used to constrain detection accuracy; sparse regularization Limiting the number of activated experts to ensure computational efficiency; modal consistency loss. The consistency between multimodal and single-modal prediction results is constrained to improve generalization ability. After training, the network in this embodiment has scene-adaptive and task-driven detection capabilities, maintaining high accuracy and low latency under different lighting and modal conditions. It can stably achieve infrared and visible light adaptive target detection in various environments such as daytime, nighttime, and complex weather.

[0049] Specifically, the joint loss function used in the model training phase is defined as: ; in, For target detection loss, For route sparsity regularization loss, For modal consistency loss, , This is the loss weighting coefficient.

[0050] Furthermore, the target detection loss is used to constrain the target detection accuracy, and it includes classification loss and location regression loss, defined as: = ; in, Used to measure the difference between the predicted class probability and the true class label. It is used to measure the deviation between the predicted candidate box location and the actual target location.

[0051] Furthermore, the classification loss, used to measure the difference between the model's predicted target class probability and the true class label, is defined as: ; in, N The number of candidate boxes or samples used in training; C The number of target categories; For the first i Each sample in category c The true label on the sample is set to 1 if the sample belongs to that category, and 0 otherwise. For the model to the first i Each sample belongs to category c The predicted probability.

[0052] Furthermore, the regression loss is used to constrain the deviation between the candidate box positions predicted by the model and the true target bounding boxes, and it is defined as: ; in, The model predicts the first i The position parameters of each candidate box correspond to the center coordinates and width and height of the candidate box, respectively; These are the corresponding real target bounding box position parameters; It is an L1 norm.

[0053] Furthermore, to avoid computational redundancy caused by activating too many expert branches during the inference phase, this embodiment introduces a routing sparsity regularization loss to constrain the distribution of expert activation weights in the gating network output, defined as follows: ; in, E For the number of expert branches, The first output of the gating network e The activation weight of each expert branch.

[0054] The embodiments of this application minimize the routing sparsity regularization loss, which makes the gating network tend to activate only a small number of expert branches under each input sample, thereby reducing the overall computational complexity and improving inference efficiency.

[0055] Furthermore, to improve the detection stability of the model in scenarios with missing modes or unbalanced mode quality, this application further introduces a mode consistency loss to constrain the consistency between prediction results of different modes or different expert branches, which is defined as: ; in, The prediction results are based on the visible light expert branch. The prediction results are based on the infrared expert branch.

[0056] This application embodiment minimizes modal consistency loss, enabling the model to learn consistent target discrimination ability under multimodal conditions, thereby improving generalization performance across scenarios and modalities.

[0057] Therefore, the detection network has scene-adaptive and task-driven detection performance, and can stably achieve dual-modal adaptive target detection in multiple environments, balancing high detection accuracy and low latency.

[0058] To facilitate a better understanding of the infrared and visible light adaptive target detection method based on selective expert routing proposed in this application, the following is a detailed explanation. Figure 2 Further explanation is needed.

[0059] like Figure 2 As shown, Figure 2 This is a flowchart of an infrared and visible light adaptive target detection method based on selective expert routing according to an embodiment of this application. The infrared and visible light adaptive target detection method based on selective expert routing includes the following steps: S201: Input infrared and visible light images, perform usability discrimination and illumination quality assessment, extract dual-light shallow layer features and generate modal confidence vectors.

[0060] S202 introduces a lightweight gating network as a routing controller, which generates selective expert activation weights based on modal confidence and content features to achieve dynamic routing decisions.

[0061] S203 feeds the dual-light features into the parallel branches of the visible light expert, infrared expert, and fusion expert respectively, and extracts modality sensitivity and semantic consistency detection features through the task-specialized structure.

[0062] S204, in the candidate box generation stage, utilizes a cross-modal cross-attention mechanism to perform regional-level weighted fusion of three types of expert features, thereby strengthening the salient areas of the target and suppressing background interference.

[0063] S205 uses routing weights to weight and integrate the classification and regression results output by expert branches to generate adaptive fusion detection output, achieving robust prediction in the case of modality loss.

[0064] S206 constructs an end-to-end joint training framework, which optimizes the network through detection loss, routing sparsity regularization and modality consistency constraints, enabling the network to have scene-adaptive and task-driven detection capabilities.

[0065] The infrared and visible light adaptive target detection method based on selective expert routing proposed in this application involves performing heteroscale shallow encoding on infrared and visible light images to obtain first and second initial multimodal features, determining modality confidence vectors, and determining target expert branches based on these vectors. The first and second initial multimodal features are then input into the target expert branches to obtain detection features. Based on a cross-modal cross-attention mechanism, region-level feature interaction is performed on the detection features to obtain candidate boxes. Finally, the target detection result is obtained by weighted ensemble based on the classification probabilities and regression results of the candidate boxes. This method solves the problems of poor modality adaptability, fusion redundancy, wasted computational resources, and insufficient detection robustness and inference efficiency in related technologies, achieving efficient and high-quality target detection.

[0066] Next, referring to the accompanying drawings, an infrared and visible light adaptive target detection device based on selective expert routing, according to an embodiment of this application, is described.

[0067] Figure 3 This is a block diagram of an infrared and visible light adaptive target detection device based on selective expert routing according to an embodiment of this application.

[0068] like Figure 3 As shown, the infrared and visible light adaptive target detection device 10 based on selective expert routing includes: an acquisition module 100, a determination module 200, and a detection module 300.

[0069] The acquisition module 100 acquires infrared images and visible light images, and performs heteroscale shallow coding on the infrared images and visible light images respectively to obtain the first initial multimodal features and the second initial multimodal features; The determination module 200 determines the modal confidence vector based on the infrared image and the visible light image, and determines at least one target expert branch based on the modal confidence vector, and inputs the first initial multimodal feature and the second initial multimodal feature into the at least one target expert branch to obtain at least one detection feature; The detection module 300, based on a preset cross-modal cross-attention mechanism, performs regional-level feature interaction on each detection feature to obtain candidate boxes for each target expert branch, and then weights and integrates the candidate boxes of each target expert branch according to their classification probability and regression results to obtain the target detection result.

[0070] According to one embodiment of this application, the determining module 200 is specifically used for: Modal availability discrimination and illumination quality assessment were performed on infrared and visible light images, respectively, to obtain modal availability discrimination results and illumination quality assessment results; Based on the modal availability discrimination results and illumination quality assessment results, the conditional vectors of the infrared image and the visible light image are determined. Modal confidence vectors are generated based on the conditional vectors of the infrared image and the visible light image.

[0071] According to one embodiment of this application, the determining module 200 is specifically used for: Determine the expert branch library, which includes visible light experts, infrared experts, and fusion experts; The modal confidence vector, infrared image, and visible light image are input to a preset routing controller, which then outputs the activation weights of the visible light expert, the infrared expert, and the fusion expert, respectively. At least one target expert branch is determined based on the activation weights of the visible light expert, the infrared expert, and the fusion expert.

[0072] According to one embodiment of this application, the weighted ensemble result of the classification probabilities of the candidate boxes is: ; in, P This is the weighted ensemble result of the classification probabilities of the candidate boxes. For the activated expert branch, For the first e Routing weights for each expert branch For the first e The classification probabilities of candidate boxes output by each expert branch.

[0073] According to one embodiment of this application, the weighted integration result of the regression results of the candidate boxes is as follows: ; in, The weighted ensemble result of the regression results for the candidate boxes. The candidate box regression result is output by the e-th expert branch.

[0074] It should be noted that the foregoing explanation of the embodiment of the infrared and visible light adaptive target detection method based on selective expert routing also applies to the infrared and visible light adaptive target detection device based on selective expert routing in this embodiment, and will not be repeated here.

[0075] The infrared and visible light adaptive target detection device based on selective expert routing proposed in this application performs heteroscale shallow encoding on infrared and visible light images to obtain first and second initial multimodal features, determines modality confidence vectors, and determines target expert branches based on the modality confidence vectors. The first and second initial multimodal features are input into the target expert branches to obtain detection features. Based on a cross-modal cross-attention mechanism, the detection features are subjected to region-level feature interaction to obtain candidate boxes. The target detection result is obtained by weighted integration based on the classification probability and regression results of the candidate boxes. This solves the problems of poor modality adaptability, fusion redundancy, waste of computational resources, and insufficient detection robustness and inference efficiency in related technologies, achieving efficient and high-quality target detection.

[0076] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0077] When the processor 402 executes the program, it implements the infrared and visible light adaptive target detection method based on selective expert routing provided in the above embodiments.

[0078] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0079] The memory 401 is used to store computer programs that can run on the processor 402.

[0080] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0081] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0082] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0083] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0084] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described infrared and visible light adaptive target detection method based on selective expert routing.

[0085] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the infrared and visible light adaptive target detection method based on selective expert routing.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0087] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0088] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An infrared and visible light adaptive target detection method based on selective expert routing, characterized in that, include: Infrared and visible light images are acquired, and heteroscale shallow coding is performed on the infrared and visible light images respectively to obtain a first initial multimodal feature and a second initial multimodal feature; A modal confidence vector is determined based on the infrared image and the visible light image, and at least one target expert branch is determined based on the modal confidence vector. The first initial multimodal feature and the second initial multimodal feature are input into the at least one target expert branch to obtain at least one detection feature. Based on a pre-defined cross-modal attention mechanism, regional feature interaction is performed on each detection feature to obtain candidate boxes for each target expert branch. The target detection result is obtained by weighted integration based on the classification probability and regression results of the candidate boxes of each target expert branch.

2. The method according to claim 1, characterized in that, The step of determining the modality confidence vector based on the infrared image and the visible light image includes: Modal availability discrimination and illumination quality assessment are performed on the infrared image and the visible light image respectively to obtain modal availability discrimination results and illumination quality assessment results; Based on the modal availability discrimination results and illumination quality assessment results, the conditional vector of the infrared image and the conditional vector of the visible light image are determined; A modal confidence vector is generated based on the conditional vector of the infrared image and the conditional vector of the visible light image.

3. The method according to claim 1, characterized in that, The step of determining at least one target expert branch based on the modality confidence vector includes: A database of expert branches is defined, which includes visible light experts, infrared experts, and fusion experts. The modal confidence vector, the infrared image, and the visible light image are input to a preset routing controller, so that the preset routing controller outputs the activation weights of the visible light expert, the infrared expert, and the fusion expert, respectively. The at least one target expert branch is determined based on the activation weights of the visible light expert, the infrared expert, and the fusion expert.

4. The method according to claim 1, characterized in that, The weighted ensemble result of the classification probabilities of the candidate boxes is: ； in, P The weighted ensemble result of the classification probabilities of the candidate boxes. For the activated expert branch, For the first e Routing weights for each expert branch For the first e The classification probabilities of candidate boxes output by each expert branch.

5. The method according to claim 1, characterized in that, The weighted ensemble result of the candidate box regression results is as follows: ； in, The weighted integration result of the regression results of the candidate boxes. For the first e The candidate box regression results output by each expert branch.

6. An infrared and visible light adaptive target detection device based on selective expert routing, characterized in that, include: The acquisition module acquires infrared images and visible light images, and performs heteroscale shallow coding on the infrared images and the visible light images respectively to obtain first initial multimodal features and second initial multimodal features; The determination module determines a modal confidence vector based on the infrared image and the visible light image, and determines at least one target expert branch based on the modal confidence vector. The first initial multimodal feature and the second initial multimodal feature are input into the at least one target expert branch to obtain at least one detection feature. The detection module, based on a preset cross-modal cross-attention mechanism, performs regional-level feature interaction on each detection feature to obtain candidate boxes for each target expert branch, and then weights and integrates the candidate boxes of each target expert branch according to their classification probability and regression results to obtain the target detection result.

7. The apparatus according to claim 6, characterized in that, The determining module is specifically used for: Modal availability discrimination and illumination quality assessment are performed on the infrared image and the visible light image respectively to obtain modal availability discrimination results and illumination quality assessment results; Based on the modal availability discrimination results and illumination quality assessment results, the conditional vector of the infrared image and the conditional vector of the visible light image are determined; A modal confidence vector is generated based on the conditional vector of the infrared image and the conditional vector of the visible light image.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the infrared and visible light adaptive target detection method based on selective expert routing as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the infrared and visible light adaptive target detection method based on selective expert routing as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the infrared and visible light adaptive target detection method based on selective expert routing as described in any one of claims 1-5.

Images