An open-vocabulary object detection method based on multi-modal fusion and dynamic expansion

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to an open-vocabulary object detection method based on multi-modal fusion and dynamic expansion, aiming to solve the problems of insufficient robustness of visual features, static multi-modal fusion, and poor flexibility of class expansion in the prior art. The method first acquires an image to be detected and text information containing class and scene description, extracts features through a visual encoder and enhances them through a robust noise reduction layer, and then generates a candidate detection frame through a full convolutional network, while extracting text features through a CLIP text encoder. Subsequently, the candidate region features are extracted, dynamically fused through an adaptive multi-modal fusion module to generate cross-modal joint features, and finally the detection frame is screened and the confidence is calculated according to the feature similarity. The model training fuses the InfoNCE loss and the cross-entropy loss to optimize the parameters, and the new class dynamic expansion can also be realized through the analysis of natural language by a generative language model. The present application significantly improves the robustness and accuracy of detection in complex scenes, optimizes the multi-modal fusion efficiency, greatly reduces the cost of new class expansion, and the model meets the real-time detection requirements after acceleration, has good engineering deployment advantages, and can be widely applied in the fields of intelligent monitoring, automatic driving and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an open vocabulary target detection method based on multimodal fusion and dynamic expansion. Background Technology

[0002] Object detection, a core research area in computer vision, is a key technological support for practical applications such as intelligent monitoring, autonomous driving, and medical image analysis. Traditional closed-set object detection models can only identify fixed categories labeled in the training set, failing to address the detection needs of unknown categories in open scenes. Therefore, open-vocabulary object detection technology has emerged, which can break through the limitation of fixed categories and achieve the detection of new categories outside the training set. It has become an important research direction for improving the generalization ability of computer vision systems and has extremely high engineering application value.

[0003] Existing open-vocabulary object detection technologies are mostly based on vision-language pre-trained models such as CLIP and ALIGN to build a basic framework, achieving the recognition of new categories by fusing visual and textual modal information. In visual feature processing, traditional encoders such as ResNet and Transformer are used to extract features, and simple feature optimization is performed with static noise reduction methods. In the multimodal fusion stage, static strategies such as feature concatenation and fixed-weight summation are generally used to allocate modal weights. For the expansion of new categories, it mainly relies on predefined text embedding vectors or additional labeled training data to achieve model adaptation for the detection of new categories.

[0004] While existing technologies have achieved the basic functions of open-vocabulary object detection, several key technical shortcomings remain to be addressed in practical applications: First, visual features lack robustness; static denoising methods cannot adaptively handle diverse noises in complex scenes such as occlusion, low light, and background noise, leading to a significant decrease in feature discriminativeness. Second, multimodal fusion lacks dynamic adaptability; static weight allocation strategies cannot adjust weights based on different modal quality conditions such as image blurring and brief text descriptions, resulting in modal information loss. Third, category expansion lacks flexibility; relying on predefined content and additional training data is not only costly but also prone to semantic bias, making it impossible to dynamically expand new categories directly through natural language descriptions. These problems collectively restrict the detection accuracy, generalization ability, and engineering application effectiveness of open-vocabulary object detection technology, becoming a key bottleneck in the development of this field. Summary of the Invention

[0005] To address the problems existing in the background art, one aspect of the present invention provides an open vocabulary target detection method based on multimodal fusion and dynamic expansion, comprising:

[0006] S1: Obtain the image to be detected and the target text description information, wherein the target text description information includes a target category description and a scene auxiliary description;

[0007] S2: Visual features are obtained by extracting features from the image to be detected through a visual encoder; visual enhancement features are obtained by denoising the visual features through a robust denoising layer;

[0008] S3: Multiple candidate detection boxes are generated by inputting visual enhancement features into a fully convolutional network for object detection;

[0009] S4: Extract target description features from the target text description information using the CLIP text encoder;

[0010] S5: Extract the candidate region features corresponding to each candidate detection box from the visual enhancement features based on the position of each candidate detection box, and dynamically fuse each candidate region feature with the target description feature through the adaptive multimodal fusion module to generate the corresponding cross-modal joint features;

[0011] S6: Select the final detection box of the target based on the similarity between the cross-modal joint features corresponding to each candidate detection box and the target description features, and calculate the confidence of the target based on the similarity between the cross-modal joint features corresponding to the final detection box of the target and the target description features.

[0012] Another aspect of the present invention provides an open vocabulary target detection system based on multimodal fusion and dynamic expansion, the system comprising a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the open vocabulary target detection method based on multimodal fusion and dynamic expansion.

[0013] Another aspect of the present invention provides a computer storage medium storing a remote monitoring program, which, when executed by a processor, implements the aforementioned open vocabulary target detection method based on multimodal fusion and dynamic expansion.

[0014] The present invention has at least the following beneficial effects

[0015] This invention significantly improves the detection accuracy and feature discriminativeness of the model in complex scenarios such as occlusion, low light, and background noise by combining a visual feature enhancement module with self-supervised contrastive learning and a robust noise reduction layer to optimize visual features. It employs an adaptive layer weight multimodal fusion mechanism, dynamically adjusting visual and text modal weights through collaborative attention and optimizing fused features by combining cross-entropy loss and contrastive loss. This fully exploits cross-modal complementary information, effectively reducing modal information loss and improving multimodal fusion efficiency. Furthermore, it leverages generative language model parsing and CLIP text encoding to achieve natural language-driven dynamic category expansion, eliminating the need for predefined text embeddings or additional training data. This significantly enhances the flexibility of category expansion, reduces expansion costs and semantic bias, and qualitatively improves the detection performance of new categories. Simultaneously, after INT8 quantization and TensorRT acceleration, this solution achieves model size compression and improved inference speed, meeting real-time detection requirements. It possesses excellent engineering deployment advantages and can be widely adapted to different hardware platforms and application scenarios. Overall, it significantly improves the accuracy, robustness, and scalability of open-vocabulary object detection, effectively addressing various key technical deficiencies of existing technologies. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0017] Figure 2 Overall architecture diagram in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the working process of the robust noise reduction layer of the present invention;

[0019] Figure 4 This is a performance comparison chart of different model configurations in a specific embodiment of the present invention;

[0020] Figure 5 The figure shows the training loss and detection metric change curves of the model in a complex scene in a specific embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please refer to Figures 1-5 One aspect of the present invention is an open vocabulary target detection method based on multimodal fusion and dynamic expansion, comprising:

[0023] S1: Obtain the image to be detected and the target text description information, wherein the target text description information includes a target category description and a scene auxiliary description;

[0024] Preferably, for the text description of a new category target input by the user, it is first parsed using a generative language model (GPT model or LLaMA model) to generate multiple structured text information; then, the structured text information is transformed into high-dimensional embedding vectors using a CLIP text encoder to obtain the target description features of the new category target.

[0025]

[0026] in, The target description features of the new category of targets; M represents the amount of structured text information; Indicates CLIP text encoder; The first target of the new category c represents the A structured text message.

[0027] In this embodiment, the image to be detected and the target text description information containing the target category description and scene auxiliary description are acquired. For the new category target text description input by the user, it is first parsed by a generative language model such as GPT or LLaMA to generate multiple structured text information. Then, the CLIP text encoder converts these structured texts into high-dimensional embedding vectors to obtain the target description features corresponding to the new category target. This step not only provides comprehensive and detection-compliant basic data support for subsequent core processes such as visual feature enhancement, candidate detection box generation, and multimodal fusion, but also realizes the transformation of natural language into machine-recognizable high-dimensional features through the collaborative processing of generative language model parsing and CLIP encoding. The effective transformation breaks through the limitations of traditional open vocabulary detection that relies on predefined text embeddings, laying a key feature foundation for the dynamic expansion of new categories. The inclusion of scene-assisted descriptions also makes the subsequent detection process more adaptable to actual application scenarios, improving the targeting of the detection. Under the trained model, the target description features of the new category can be directly matched with the candidate detection boxes of the image to be detected, as in steps S5 to S6, without the need to retrain the model. Here, the target category describes the type of target to be detected, and the scene-assisted description describes the scene information of the image to be detected, such as low-light outdoor nighttime environment, strong light direct midday outdoor scene, crowded shopping mall partially occluded scene, and rainy outdoor scene with rain noise, etc.

[0028] S2: Visual features are obtained by extracting features from the image to be detected through a visual encoder; visual enhancement features are obtained by denoising the visual features through a robust denoising layer;

[0029] Preferably, the visual encoder uses a ResNet-50 or Swing Transformer network to extract visual features from the detection image; the robust denoising layer includes a denoising module, a weight generation module, and a reweighting module; wherein, the denoising module is used to calculate the local noise level of each pixel in the visual features, and assigns pixels with local noise levels greater than a set threshold. Pixels that do not meet the criteria are identified as noise pixels, while those that do not are marked as valid target pixels. The weight generation module is used to generate a channel weight matrix by sequentially passing the visual features of the detected image through a GAP layer, a first FC layer, a ReLU activation function, a second FC layer, and a Sigmoid activation function. The reweighting module is used to apply attenuation coefficients... The pixels marked as noise pixels in the visual features are subjected to feature value attenuation processing, while the pixels marked as valid target pixels in the visual features remain unchanged, resulting in pixel-level suppressed visual features. The pixel-level suppressed visual features are then multiplied channel by channel weight matrix to obtain visual enhancement features.

[0030] Preferably, the local noise level of each pixel in the computational visual features includes:

[0031]

[0032] in, Representing pixels in visual features The local noise level; Representing pixels in visual features The neighborhood; Representing pixels in visual features Pixel values; Representing pixels in visual features neighboring pixels The pixel value.

[0033] In this embodiment, ResNet-50 or SwinTransformer is first used as a visual encoder to extract features from the image to be detected, obtaining basic visual features. Then, a robust denoising layer consisting of a denoising module, a weight generation module, and a reweighting module is used to optimize the basic visual features. First, the local noise level of each pixel is calculated to distinguish between noisy pixels and effective target pixels. Then, a channel weight matrix is ​​generated, and the feature values ​​of noisy pixels are attenuated by an attenuation coefficient. The pixel-level suppressed visual features are multiplied with the channel weight matrix channel by channel to finally obtain the visually enhanced features. This step abandons the traditional static feature denoising and extraction method, and can adaptively identify and process various types of noise in complex scenes. It effectively enhances the discriminativeness and stability of visual features, and greatly improves the robustness of visual features in complex scenes such as occlusion, low light, and background noise. It provides high-quality visual feature support for the accurate generation of subsequent candidate detection boxes, reduces the interference of noise on the detection results from the source, and improves the basic feature quality of the overall detection process.

[0034] S3: Multiple candidate detection boxes are generated by inputting visual enhancement features into a fully convolutional network for object detection;

[0035] Preferably, the fully convolutional network includes: three consecutive 3×3 convolutional layers, one 1×1 convolutional layer, and parallel probability branch output layers and coordinate branch output layers; wherein, the three consecutive 3×3 convolutional layers and one 1×1 convolutional layer are used to extract features from the input feature map; the probability branch output layer includes a 1×1 convolution and a Sigmoid activation function, used to output the probability of the target's presence; the coordinate branch output layer includes a 1×1 convolution and a Sigmoid activation function, the 1×1 convolutional layer directly outputs the detection box coordinate parameters corresponding to each spatial location of the feature map, then normalizes them to the [0,1] interval using the Sigmoid function, and then... ; The pixel coordinates mapped to the image to be detected; where, and The width and height of the image to be detected; , , and These are the normalized coordinates of the top and bottom corners of the detection box. , , and The coordinates of the top and bottom corners of the mapped detection box are set; the validity range of the coordinates and the confidence threshold are set, and the detection boxes whose coordinates are within the validity range and whose confidence threshold is greater than the set threshold are retained as candidate detection boxes.

[0036] In this embodiment, the obtained visual enhancement features are input into a customized fully convolutional network to perform target detection and generate candidate detection boxes. This fully convolutional network includes three consecutive 3×3 convolutional layers, one 1×1 convolutional layer, and parallel probability branch output layers and coordinate branch output layers. First, the convolutional layers perform feature extraction. Then, the probability branch outputs the target presence probability, and the coordinate branch normalizes the output coordinate parameters to the pixel coordinates of the image to be detected using a sigmoid function. Finally, a coordinate validity range and confidence threshold are set to filter out qualified detection boxes as candidate detection boxes. This step is based on visual enhancement features optimized with robust noise reduction. The bounding box generation process utilizes a customized fully convolutional network structure to more accurately extract target location information from visual features. The parallel dual-branch design enables the synchronous and accurate output of target existence probability and coordinate parameters. Coordinate normalization and pixel mapping ensure a high degree of matching between the detection box coordinates and the original image. Threshold filtering effectively eliminates invalid and low-confidence detection boxes, providing an accurate and effective set of candidate detection boxes for subsequent candidate region feature extraction and multimodal fusion. This reduces the interference of invalid boxes on subsequent processes from the source, improves the efficiency of the overall detection process, and also reduces the probability of missed detections and false detections, laying a solid foundation for the accurate selection of the final detection boxes.

[0037] S4: Extract target description features from the target text description information using the CLIP text encoder;

[0038] S5: Extract the candidate region features corresponding to each candidate detection box from the visual enhancement features based on the position of each candidate detection box, and dynamically fuse each candidate region feature with the target description feature through the adaptive multimodal fusion module to generate the corresponding cross-modal joint features;

[0039] Preferably, the step of dynamically fusing the features of each candidate region with the cross-modal joint features through an adaptive multimodal fusion module includes:

[0040] S51: The target description features are mapped to the same dimension as the candidate region features through a 1×1 convolution. The mapped target description features are then mapped to query vectors through three linear layers. Key vector Value vector ;

[0041] S52: Map the candidate region features into query vectors through three linear layers. Key vector Sum value vector ;

[0042] S53: Calculate cross-modal attention weights and :

[0043]

[0044]

[0045] in, This represents the activation function. Indicates transpose; Indicates the scaling factor;

[0046] S54: Based on cross-modal attention weights and For value vectors respectively Sum value vector Visual interaction features are obtained by weighting. Text interaction features :

[0047]

[0048]

[0049] S55: Visual interaction features Text interaction features After element-wise addition, the input is processed by a multi-scale convolution module to perform feature aggregation and enhancement, resulting in cross-modal joint features.

[0050] In this embodiment, candidate region features are first extracted from the visual enhancement features based on the positions of each candidate detection box. Then, each candidate region feature and the target description feature are fed into the adaptive multimodal fusion module to complete dynamic fusion. This module first maps the target description feature to the same dimension as the candidate region feature through a 1×1 convolution. Then, the two are passed through three linear layers to generate corresponding query, key, and value vectors. The bidirectional cross-modal attention weights from text to vision and from vision to text are calculated. The corresponding value vectors are weighted using the weights to obtain visual interaction features and text interaction features. Finally, the two types of interaction features are added element-wise and then input into the multi-scale hybrid convolution module for feature aggregation. The enhancement process ultimately generates cross-modal joint features. This step achieves dynamic adaptive adjustment of visual and text modal weights through bidirectional cross-modal attention, which can mine complementary information based on the quality of modal information and solve the problem of poor fusion effect when a single modality is damaged. The multi-scale hybrid convolution module further aggregates and enhances the fused features, effectively improving the representation ability of cross-modal joint features. This allows the features to simultaneously possess richer visual spatial location information, text semantic category information, and multi-scale feature information, providing higher quality feature support for subsequent calculation of feature similarity and selection of final detection boxes, and greatly improving the accuracy and reliability of subsequent detection steps.

[0051] S6: Select the final detection box of the target based on the similarity between the cross-modal joint features corresponding to each candidate detection box and the target description features, and calculate the confidence of the target based on the similarity between the cross-modal joint features corresponding to the final detection box of the target and the target description features.

[0052] Preferably, step S6 includes:

[0053] S61: Map the cross-modal joint features corresponding to each candidate detection box to the same dimension as the target description features through a multilayer perceptron, and calculate the matching degree between each candidate detection box and the target description features through cosine similarity;

[0054] S62: Input the matching degree between each candidate detection box and the target description feature into Softmax to calculate the confidence of the target corresponding to each candidate box;

[0055] S63: Retain the candidate detection box with the highest confidence corresponding to the target, and remove the candidate detection boxes with an overlap of more than a set threshold to obtain the final detection box corresponding to the target;

[0056] S64: Integrate the final detection box coordinates, target category, detection box confidence, and target category confidence to output the target detection result.

[0057] In this embodiment, the cross-modal joint features corresponding to each candidate detection box are first mapped to the same dimension as the target description features using a multilayer perceptron. Cosine similarity is then used to calculate the matching degree between the two. The matching degree is then input into a softmax function to obtain the confidence score of the target corresponding to each candidate box. Subsequently, the candidate detection box with the highest confidence score is retained, and candidate boxes with an overlap exceeding a set threshold are removed to determine the final detection box of the target. Finally, the coordinates of the final detection box, the target category, the detection box confidence score, and the target category confidence score are integrated to output a complete target detection result. This step uses cross-modal joint features that integrate visual spatial location information and textual semantic category information as the basis for judgment. Cosine similarity can accurately measure the semantic and spatial matching degree between features, the Softmax function makes the confidence calculation more reasonable and discriminative, and the overlapping box removal operation effectively avoids the problems of duplicate detection and false detection, reducing the interference of redundant boxes on the detection results. The output format that integrates multi-dimensional information makes the detection results more complete and intuitive, which can meet the needs of practical applications. As the final step of the entire detection process, this step ensures the uniqueness and accuracy of the final detection box through precise screening and quantitative judgment at each level, which greatly improves the overall accuracy and reliability of open vocabulary target detection, and makes the detection results highly consistent with the target requirements of the text description.

[0058] Preferably, during model training, the model refers to the network model used to perform steps S1 to S6, which generates positive and negative sample pairs of the image to be detected through feature enhancement processing, constructs the InfoNCE loss function based on the features extracted from the positive and negative sample pairs by the visual encoder, constructs the cross-entropy loss based on the true label and prediction result corresponding to the cross-modal joint features of each candidate detection box, constructs the total loss function of the model based on the InfoNCE loss function and the cross-entropy loss, and updates the parameters of the model based on the total loss function.

[0059] In this embodiment, the InfoNCE loss function includes:

[0060]

[0061] in, For the sample Embedded vector, For positive sample embedding vectors (different augmented samples of the same image). The negative sample embedding vector (features of different images). This is the cosine similarity calculation function. The temperature parameter has a value range of 0.05-0.2; the different enhanced samples of the same image include: positive samples obtained by randomly cropping, color perturbation, and rotation of the image;

[0062] In this embodiment, the cross-entropy loss function includes:

[0063]

[0064] in, The total number of categories, For the sample The true label (one-hot encoded form). The class probability distribution predicted by the model. This is a cross-modal fusion feature.

[0065] In this embodiment, the total loss function of the model includes:

[0066]

[0067] in, For cross-entropy loss, To compare the losses, This is the balance coefficient, with a value ranging from 0.3 to 0.7.

[0068] In this embodiment, positive and negative sample pairs are generated for the image to be detected through feature enhancement processing. An InfoNCE loss function is constructed based on the features extracted from these positive and negative sample pairs by the visual encoder. Then, a cross-entropy loss is constructed based on the true label and prediction results of the cross-modal joint features corresponding to each candidate detection box. Subsequently, the InfoNCE loss function and the cross-entropy loss are fused to build the model's total loss function, and the model parameters are iteratively updated based on the total loss function. This step overcomes the limitations of training with a single loss function, achieving a balance between different training objectives through the synergistic optimization of two loss functions. The InfoNCE loss enhances the discriminativeness and robustness of visual features, significantly improving the model's learning and generalization abilities for image features in complex scenes. The cross-entropy loss precisely optimizes the model's accuracy in classifying target categories. This allows the model to simultaneously achieve the dual training objectives of visual feature optimization and cross-modal category matching during training, resulting in more stable model convergence. This effectively improves the overall detection accuracy of the model in both conventional and complex scenes, and lays a solid foundation of model parameters for the accuracy of new category detection, ensuring the efficiency and accuracy of subsequent detection processes.

[0069] Please see Figure 4 and Figure 5 To verify the effectiveness of this invention, experiments were conducted on publicly available datasets such as COCO, VOC, and OpenImagesV6. The datasets included 118k training images, 5k validation images, and 30% more complex scene data (10% occlusion + 10% noise + 10% low illumination) to construct a test set (2000 images) for 10 new categories. Experimental results show that: in normal scenes, the model of this invention achieves an accuracy of 85% and an mAP of 80%, an improvement of more than 25% compared to the baseline model; in complex scenes, the model achieves an accuracy of 75% and an F1-score of 72.5%, significantly outperforming existing technologies; the new category detection accuracy reaches 68.9%, a significant improvement compared to the baseline model (<10%); and the model inference speed reaches 35fps, meeting the requirements for real-time detection.

[0070] Another aspect of the present invention provides an open vocabulary target detection system based on multimodal fusion and dynamic expansion, the system comprising a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the open vocabulary target detection method based on multimodal fusion and dynamic expansion.

[0071] Another aspect of the present invention provides a computer storage medium storing a remote monitoring program, which, when executed by a processor, implements the aforementioned open vocabulary target detection method based on multimodal fusion and dynamic expansion.

[0072] To verify the effectiveness of this invention, experiments were conducted on publicly available datasets such as COCO, VOC, and OpenImagesV6. The datasets included 118k training images, 5k validation images, and 30% more complex scene data (10% occlusion + 10% noise + 10% low illumination) to construct a test set (2000 images) for 10 new categories. Experimental results show that: in normal scenes, the model of this invention achieves an accuracy of 85% and an mAP of 80%, an improvement of more than 25% compared to the baseline model; in complex scenes, the model achieves an accuracy of 75% and an F1-score of 72.5%, significantly outperforming existing technologies; the new category detection accuracy reaches 68.9%, a significant improvement compared to the baseline model (<10%); and the model inference speed reaches 35fps, meeting the requirements for real-time detection.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] In summary, this invention significantly improves the detection accuracy and feature discriminativeness of the model in complex scenarios such as occlusion, low light, and background noise by combining a visual feature enhancement module with self-supervised contrastive learning and a robust noise reduction layer to optimize visual features. It employs an adaptive layer weight multimodal fusion mechanism, dynamically adjusting visual and text modal weights through collaborative attention and optimizing fused features by combining cross-entropy loss and contrastive loss. This fully exploits cross-modal complementary information, effectively reducing modal information loss and improving multimodal fusion efficiency. Furthermore, it leverages generative language model parsing and CLIP text encoding to achieve natural language-driven dynamic category expansion, eliminating the need for predefined text embeddings or additional training data. This significantly enhances the flexibility of category expansion, reduces expansion costs and semantic bias, and qualitatively improves the detection performance of new categories. Simultaneously, after INT8 quantization and TensorRT acceleration, this solution achieves model size compression and improved inference speed, meeting real-time detection requirements. It possesses excellent engineering deployment advantages and can be widely adapted to different hardware platforms and application scenarios. Overall, it significantly improves the accuracy, robustness, and scalability of open-vocabulary object detection, effectively addressing various key technical deficiencies of existing technologies.

[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An open vocabulary target detection method based on multimodal fusion and dynamic expansion, characterized in that, include: S1: Obtain the image to be detected and the target text description information, wherein the target text description information includes a target category description and a scene auxiliary description; S2: Visual features are obtained by extracting features from the image to be detected through a visual encoder; visual enhancement features are obtained by denoising the visual features through a robust denoising layer; S3: Multiple candidate detection boxes are generated by inputting visual enhancement features into a fully convolutional network for object detection; S4: Extract target description features from the target text description information using the CLIP text encoder; S5: Extract the candidate region features corresponding to each candidate detection box from the visual enhancement features based on the position of each candidate detection box, and dynamically fuse each candidate region feature with the target description feature through the adaptive multimodal fusion module to generate the corresponding cross-modal joint features; S6: Select the final detection box of the target based on the similarity between the cross-modal joint features corresponding to each candidate detection box and the target description features, and calculate the confidence of the target based on the similarity between the cross-modal joint features corresponding to the final detection box of the target and the target description features.

2. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 1, characterized in that, The visual encoder uses a ResNet-50 or Swing Transformer network to extract visual features from the detection image; the robust noise reduction layer includes a noise reduction module, a weight generation module, and a reweighting module; wherein, the noise reduction module is used to calculate the local noise level of each pixel in the visual features, and assigns pixels with local noise levels greater than a set threshold. Pixels that do not meet the criteria are identified as noise pixels, while those that do not are marked as valid target pixels. The weight generation module is used to generate a channel weight matrix by sequentially passing the visual features of the detected image through a GAP layer, a first FC layer, a ReLU activation function, a second FC layer, and a Sigmoid activation function. The reweighting module is used to apply attenuation coefficients... The pixels marked as noise pixels in the visual features are subjected to feature value attenuation processing, while the pixels marked as valid target pixels in the visual features remain unchanged, resulting in pixel-level suppressed visual features. The pixel-level suppressed visual features are then multiplied channel by channel weight matrix to obtain visual enhancement features.

3. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 2, characterized in that, The local noise level of each pixel in the computational visual features includes: in, Representing pixels in visual features The local noise level; Representing pixels in visual features ; Representing pixels in visual features Pixel values; Representing pixels in visual features neighboring pixels The pixel value.

4. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 1, characterized in that, The fully convolutional network comprises: three consecutive 3×3 convolutional layers, one 1×1 convolutional layer, and parallel probability branch output layers and coordinate branch output layers. The three consecutive 3×3 convolutional layers and the one 1×1 convolutional layer are used for feature extraction from the input feature map. The probability branch output layer includes a 1×1 convolution and a Sigmoid activation function to output the probability of target presence. The coordinate branch output layer includes a 1×1 convolution and a Sigmoid activation function. The 1×1 convolutional layer directly outputs the coordinate parameters of the detection box corresponding to each spatial location of the feature map, which are then normalized to the [0,1] interval using the Sigmoid function, and then... ; The pixel coordinates mapped to the image to be detected; where, and The width and height of the image to be detected; , , and These are the normalized coordinates of the top and bottom corners of the detection box. , , and The coordinates of the top and bottom corners of the mapped detection box are set; the validity range of the coordinates and the confidence threshold are set, and the detection boxes whose coordinates are within the validity range and whose confidence threshold is greater than the set threshold are retained as candidate detection boxes.

5. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 1, characterized in that, The dynamic fusion of each candidate region feature with cross-modal joint features via an adaptive multimodal fusion module includes: S51: The target description features are mapped to the same dimension as the candidate region features through a 1×1 convolution. The mapped target description features are then mapped to query vectors through three linear layers. Key vector Value vector ; S52: Map the candidate region features into query vectors through three linear layers. Key vector Sum value vector ; S53: Calculate cross-modal attention weights and : in, This represents the activation function. Indicates transpose; Indicates the scaling factor; S54: Based on cross-modal attention weights and For value vectors respectively Sum value vector Visual interaction features are obtained by weighting. Text interaction features : S55: Visual interaction features Text interaction features After element-wise addition, the input is given to a multi-scale hybrid convolution module for feature aggregation and enhancement to obtain cross-modal joint features.

6. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 1, characterized in that, Step S6 includes: S61: Map the cross-modal joint features corresponding to each candidate detection box to the same dimension as the target description features through a multilayer perceptron, and calculate the matching degree between each candidate detection box and the target description features through cosine similarity; S62: Input the matching degree between each candidate detection box and the target description feature into Softmax to calculate the confidence of the target corresponding to each candidate box; S63: Retain the candidate detection box with the highest confidence corresponding to the target, and remove the candidate detection boxes with an overlap of more than a set threshold to obtain the final detection box corresponding to the target; S64: Integrate the final detection box coordinates, target category, detection box confidence, and target category confidence to output the target detection result.

7. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 1, characterized in that, During model training, the model refers to the network model used to perform steps S1 to S6. Positive and negative sample pairs of the image to be detected are generated through feature enhancement processing. The InfoNCE loss function is constructed based on the features extracted from the positive and negative sample pairs by the visual encoder. Cross-entropy loss is constructed based on the true label and prediction result corresponding to the cross-modal joint features of each candidate detection box. The total loss function of the model is constructed based on the InfoNCE loss function and the cross-entropy loss. The parameters of the model are updated based on the total loss function.

8. The open vocabulary target detection method based on multimodal fusion and dynamic expansion according to claim 1, characterized in that, For the text description of a new category target input by the user, it is first parsed using a generative language model (GPT or LLaMA) to generate multiple structured text information. Then, the CLIP text encoder transforms the structured text information into high-dimensional embedding vectors to obtain the target description features of the new category target. in, The target description features of the new category of targets; M represents the amount of structured text information; Indicates CLIP text encoder; The first target of the new category c represents the A structured text message.

9. An open vocabulary target detection system based on multimodal fusion and dynamic expansion, characterized in that, The system includes a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the open vocabulary target detection method based on multimodal fusion and dynamic expansion as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores a remote monitoring program, which, when executed by a processor, implements an open vocabulary target detection method based on multimodal fusion and dynamic expansion as described in any one of claims 1 to 8.

Images