An Open Vocabulary Target Detection Method and System Based on Hybrid Expert Fine-tuning
By introducing a hybrid expert module into the Grounding DINO model and fine-tuning it, the application problem of hybrid expert models in open vocabulary object detection is solved, and the performance of the model on open vocabulary detection tasks is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2025-05-21
- Publication Date
- 2026-07-17
AI Technical Summary
There is currently no effective method to incorporate hybrid expert models into the Grounding DINO model to achieve efficient open vocabulary target detection.
The feedforward neural network module in the decoder of the Grounding DINO model is replaced with the first hybrid expert module, and each linear layer of the image-text cross-attention module and the text-image cross-attention module is replaced with the second hybrid expert module. The hybrid expert DINO model is fine-tuned using an image-text pair dataset, other parameters are frozen, and a sparse routing mechanism is used for training.
It improves the model's zero-shot transfer capability on open vocabulary detection tasks and enhances its detection performance on the COCO, LVIS-minival, and LVIS-val datasets.
Smart Images
Figure CN120471106B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of open vocabulary target detection, and specifically to an open vocabulary target detection method and system based on hybrid expert fine-tuning. Background Technology
[0002] In recent years, open-vocabulary object detection has attracted widespread attention and gradually become a research hotspot. Unlike tasks that train on a specific set of categories, open-vocabulary object detection aims to detect objects in an image by specifying them as text input.
[0003] Grounding DINO is designed for open-world multimodal object detection. It takes both images and text as input and outputs the location of the image target indicated by the text. Through pre-training with massive amounts of data, it can achieve zero-shot generalization in open world environments.
[0004] In recent years, Hybrid Expert Models (MoEs) have been widely applied in the fields of Large Language Models (LLMs) and Multimodal Large Models (MLLMs). The Scaling Law states that model size is one of the key factors in improving model performance; with limited computational resources, training a larger model with fewer training steps is often more effective than training a smaller model with more steps. A significant advantage of MoEs is their ability to increase model size while maintaining inference speed, thereby enhancing performance. MoEs mainly consist of two key parts: sparse MoE layers, which replace the feedforward network (FFN) layers in traditional Transformer models. Each MoE layer contains several experts, each an independent neural network. The other part is a gating network or routing component, which guides the input (token) to one or more designated expert networks. The token routing method is a crucial aspect of MoE deployment; typically, the routing network has learnable parameters and is trained simultaneously with the other parts of the network.
[0005] However, there is currently no solution for how to introduce MoE into Grounding DINO to enable efficient open vocabulary target detection. Summary of the Invention
[0006] The purpose of this invention is to solve the technical problems existing in the prior art and to provide an open vocabulary target detection method and system based on hybrid expert fine-tuning.
[0007] The specific technical solution adopted in this invention is as follows:
[0008] In a first aspect, the present invention provides an open vocabulary target detection method based on hybrid expert fine-tuning, comprising:
[0009] S1. Using the Grounding DINO model as a pre-trained model for open vocabulary target detection, the feedforward neural network module in the decoder is replaced with the first hybrid expert module. At the same time, each linear layer inside the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module is replaced with the second hybrid expert module, thus obtaining the hybrid expert DINO model.
[0010] S2. Treat all parameters of the hybrid expert modules as learnable parameters, freeze all other parameters, use the sum of the loss function of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules as the total loss, fine-tune the hybrid expert DINO model with training data consisting of image-text pairs, and use the fine-tuned hybrid expert DINO model for open vocabulary target detection.
[0011] As a preferred embodiment of the first aspect, the first hybrid expert module consists of multiple first experts and a first router. Each first expert is obtained by copying the original feedforward neural network module in the decoder. After receiving the module input, the first router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.
[0012] As a preferred embodiment of the first aspect, the routing mechanism in the first router adopts a sparse routing mechanism.
[0013] As a preferred embodiment of the first aspect, the second hybrid expert module consists of multiple second experts and a second router. Each second expert is obtained by replicating the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the active expert according to the routing mechanism, and the outputs of each active expert are merged to generate the final module output.
[0014] As a preferred embodiment of the first aspect mentioned above, the routing mechanism in the second router adopts a sparse routing mechanism.
[0015] As a preferred embodiment of the first aspect above, in the image-text cross-attention module and the text-image cross-attention module, each of the query linear layer, key linear layer, value linear layer and output linear layer within the two modules needs to be replaced with an independent second hybrid expert module.
[0016] As a preferred embodiment of the first aspect, the number of experts in both the first hybrid expert module and the second hybrid expert module is set to 8.
[0017] Secondly, the present invention provides an open vocabulary target detection system based on hybrid expert fine-tuning, comprising:
[0018] The network building module is used to use the Grounding DINO model as a pre-trained model for open vocabulary object detection. It replaces the feedforward neural network module in the decoder with the first hybrid expert module, and replaces each linear layer inside the image-text cross attention module and the text-image cross attention module in the feature enhancement module with the second hybrid expert module, thus obtaining the hybrid expert DINO model.
[0019] The network fine-tuning module is used to treat all parameters of the hybrid expert modules as learnable parameters, freeze all other parameters, use the loss function of the Grounding DINO model itself and the sum of the load balancing loss of all hybrid expert modules as the total loss, fine-tune the hybrid expert DINO model with training data consisting of image-text pairs, and use the fine-tuned hybrid expert DINO model for open vocabulary target detection.
[0020] Thirdly, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the open vocabulary target detection method based on hybrid expert fine-tuning as described in any of the solutions of the first aspect above.
[0021] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;
[0022] The memory is used to store computer programs;
[0023] The processor is configured to, when executing the computer program, implement the open vocabulary target detection method based on hybrid expert fine-tuning as described in any of the first aspects above.
[0024] Compared with the prior art, the present invention has the following advantages:
[0025] This invention proposes an open-vocabulary object detection method based on hybrid expert fine-tuning. This method introduces a hybrid expert model to enhance the model's expressive power, building upon traditional open-vocabulary object detection algorithms. For the first time, this invention modifies the decoder, transforming the original FFN module into a hybrid expert system composed of multiple FFNs. Different expert combinations can process different tokens, improving the model's open-vocabulary detection capability. To accelerate training, this invention employs fine-tuning, using image and text datasets to fine-tune the sparse MoE layer parameters based on pre-trained Grounding DINO, while freezing all other network parameters. Experiments on the COCO, LVIS-minival, and LVIS-val datasets demonstrate that this invention improves the model's zero-shot transfer AP metric on these datasets, indicating an enhanced open-vocabulary detection capability. Furthermore, this method can be used as a plugin to integrate into other Transformer architectures without incurring excessive inference overhead. Attached Figure Description
[0026] Figure 1 This is a schematic diagram illustrating the steps of an open vocabulary target detection method based on hybrid expert fine-tuning.
[0027] Figure 2 This is a diagram showing the module composition of an open vocabulary target detection system based on hybrid expert fine-tuning.
[0028] Figure 3 This is a schematic diagram of the structure of a computer electronic device;
[0029] Figure 4 A schematic diagram of the structure after setting the first hybrid expert module in the decoder;
[0030] Figure 5 A schematic diagram of the structure after setting the second hybrid expert module in the feature enhancement module. Detailed Implementation
[0031] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in various embodiments of the present invention can be combined accordingly without mutual conflict.
[0032] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.
[0033] like Figure 1 As shown, in a preferred embodiment of the present invention, an open vocabulary target detection method based on hybrid expert fine-tuning is provided, which includes:
[0034] S1. Using the Grounding DINO model as a pre-trained model for open vocabulary object detection, the feedforward neural network module (FFN) in the decoder layer is replaced with the first hybrid expert module. At the same time, each linear layer inside the image-to-text cross-attention module and the text-to-image cross-attention module in the feature enhancer is replaced with the second hybrid expert module, thus obtaining the hybrid expert DINO model.
[0035] S2. Treat all parameters of the hybrid expert modules as learnable parameters, freeze all other parameters, use the sum of the loss function of the Grounding DINO model itself and the load balance loss (LBL) of all hybrid expert modules as the total loss, fine-tune the hybrid expert DINO model with training data consisting of image-text pairs, and use the fine-tuned hybrid expert DINO model for open vocabulary target detection.
[0036] It should be noted that the Grounding DINO model itself is existing technology. The Grounding DINO network structure comprises four key components: an image encoder (Image Backbone), a text encoder (Text Backbone), a feature enhancer, a language-guided query selection module, and a cross-modality decoder. Specifically, Grounding DINO uses SwinTransformer for the image backbone and BERT for the text backbone. The feature enhancer uses Deformable Self-Attention to enhance multi-scale image features and Self-Attention to enhance text features. Referring to GLIP, it adds Image-to-text Cross-Attention and Text-to-image Cross-Attention modules to help align features from different modalities. Finally, it obtains updated image / text features through a fully connected neural network. To guide text-based object detection, a language-guided query selection mechanism is designed, selecting features more relevant to the text as the decoder's query. The Cross-Modality Decoder takes a cross-modal query as input, uses Self-Attention to enhance features, and combines updated image and text features. It combines Image Cross-Attention with image features and Text Cross-Attention with text features, finally obtaining the final features for generating detection boxes through a fully connected neural network. L1 loss and GIOU loss are used for box regression. Referring to GLIP, a contrastive loss (specifically focal loss) is used to constrain whether the predicted target matches the text. Therefore, the loss function used for training the Grounding DINO model is Loss = Loss focal +Loss L1 +Loss giou Loss focal Loss L1 and Loss giouThese are the focal length loss, L1 loss, and GIOU loss, respectively. For the specific network structure, loss function, and technical details of the Grounding DINO model, please refer to the existing technical literature: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-SetObject Detection (https: / / arxiv.org / pdf / 2303.05499), which will not be elaborated here.
[0037] It should be noted that the first hybrid expert module consists of multiple first experts and a first router. Each first expert is obtained by copying the original feedforward neural network module in the decoder. That is, each first expert is an FFN. After receiving the module input, the first router routes the module input to the activation experts according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.
[0038] It should be noted that the routing mechanism in the first router can be selected according to actual needs. Theoretically, attention mechanism, soft routing, hard routing, or sparse routing are all acceptable. In the embodiments of the present invention, the routing mechanism in the first router preferably adopts the sparse routing mechanism, which is a combination of soft routing and hard routing. It retains the advantages of dynamic allocation and improves efficiency by limiting the number of participating experts. In the sparse routing mechanism, a fully connected layer can be used as the first router. Assuming that the module input of the first hybrid expert module is X1, the module input X1 can be mapped through the fully connected layer to the routing probability of each first expert being activated. The k first experts with the highest routing probabilities are selected as activated experts in a Top-k manner. Each activated expert can calculate the expert output o based on the module input X1. i =FFN i (X1), FFN i () represents the i-th activation expert in the first hybrid expert module. The routing probabilities of all k activation experts are normalized by Softmax and used as weights, which are then used to weight the outputs of all k activation experts. i A weighted sum is performed to generate the final module output o1 of the first hybrid expert module. This final module output o1 will replace the output of the original feedforward neural network module FFN in the Grounding DINO decoder and be input into subsequent cascaded Grounding DINO network layers.
[0039] Similarly, the second hybrid expert module consists of multiple second experts and a second router. Each second expert is obtained by replicating the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the active expert according to the routing mechanism, and the outputs of each active expert are merged to generate the final module output.
[0040] It should be noted that the routing mechanism in the second router can be selected according to actual needs. Theoretically, attention mechanism, soft routing, hard routing, or sparse routing are all acceptable. In the embodiments of this invention, the routing mechanism in the second router preferably adopts a sparse routing mechanism. In the sparse routing mechanism, a fully connected layer can be used as the second router. Assuming the module input of the second hybrid expert module is X2, the module input X2 can be mapped through the fully connected layer to the routing probability of each second expert being activated. The k second experts with the highest routing probabilities are selected as activation experts in a Top-k manner. Each activation expert can calculate the expert output o based on the module input X2. i =Linear i (X2), Linear i () represents the i-th activation expert of the second hybrid expert module. The routing probabilities of all k activation experts are normalized by Softmax and used as weights, which are then used to weight the outputs of all k activation experts. i A weighted sum is performed to generate the final module output o2 of the second hybrid expert module. This final module output o2 will replace the output of the original linear layer in the Grounding DINO decoder and be input into subsequent cascaded Grounding DINO network layers.
[0041] It is important to note that both the image-text cross-attention module and the text-image cross-attention module contain four linear layers: q, k, v, and out, which are respectively denoted as the query linear layer, key linear layer, value linear layer, and output linear layer. Each linear layer needs to be replaced by a second hybrid expert module, and the second hybrid expert modules formed by replacing different linear layers are independent of each other.
[0042] It should also be noted that the number of experts in the first and second hybrid expert modules can be optimized according to actual needs. In the embodiments of the present invention, the number of experts in both the first and second hybrid expert modules is preferably set to 8.
[0043] This invention introduces a hybrid expert module into the existing FFN module in the Decoder and the Cross-Attention module in the Encoder, enabling different expert combinations to process tokens with different semantic information. This enhances the network's open vocabulary detection capability. This invention is a fine-tuning method; based on the pre-trained model, the decoder is modified, and fine-tuning is performed only on the hybrid expert module in the modified decoder. This method can also be used as a plugin to add to other Transformer structures without incurring excessive inference overhead.
[0044] It should be noted that the method steps shown in S1 to S2 above can essentially be implemented in the form of computer programs or software functional modules.
[0045] Therefore, based on the same inventive concept, such as Figure 2 As shown, the present invention also provides an open vocabulary target detection system based on hybrid expert fine-tuning, corresponding to the open vocabulary target detection method based on hybrid expert fine-tuning provided in the above embodiments, which includes:
[0046] The network building module is used to use the Grounding DINO model as a pre-trained model for open vocabulary object detection. It replaces the feedforward neural network module in the decoder with the first hybrid expert module, and replaces each linear layer inside the image-text cross attention module and the text-image cross attention module in the feature enhancement module with the second hybrid expert module, thus obtaining the hybrid expert DINO model.
[0047] The network fine-tuning module is used to treat all parameters of the hybrid expert modules as learnable parameters, freeze all other parameters, use the loss function of the Grounding DINO model itself and the sum of the load balancing loss of all hybrid expert modules as the total loss, fine-tune the hybrid expert DINO model with training data consisting of image-text pairs, and use the fine-tuned hybrid expert DINO model for open vocabulary target detection.
[0048] Furthermore, based on the same inventive concept, such as Figure 3 As shown, the present invention also provides a computer electronic device corresponding to the open vocabulary target detection method based on hybrid expert fine-tuning provided in the above embodiments, which includes a memory and a processor;
[0049] The memory is used to store computer programs;
[0050] The processor is configured to implement the open vocabulary target detection method based on hybrid expert fine-tuning as described above when executing the computer program;
[0051] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.
[0052] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to the open vocabulary target detection method based on hybrid expert fine-tuning. The storage medium stores a computer program, which, when executed by a processor, can realize the open vocabulary target detection method based on hybrid expert fine-tuning as described above.
[0053] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can realize the open vocabulary target detection method based on hybrid expert fine-tuning as described above.
[0054] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor and can perform the aforementioned steps S1 to S2.
[0055] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.
[0056] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0057] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.
[0058] The present invention will further demonstrate the detailed implementation process and technical effects of the open vocabulary target detection method based on hybrid expert model fine-tuning shown in steps S1 to S2 on a specific dataset through a specific embodiment, so as to facilitate understanding of the essence of the present invention.
[0059] Example
[0060] This embodiment refers to the open vocabulary target detection method based on hybrid expert model fine-tuning shown in steps S1 to S2 above, and demonstrates the model construction and fine-tuning process of the method on a specific dataset. It also shows some specific parameter settings and implementation results.
[0061] In this embodiment, the open vocabulary target detection method based on fine-tuning of a hybrid expert model has the following specific steps:
[0062] Step 1.1: For the development of vocabulary target detection, the Grounding DINO model is used as a pre-trained model, and pre-trained weights are loaded.
[0063] Step 1.2: Sparsify the FFN module in the Grounding DINO decoder. Specifically, duplicate the FFN model in the pre-trained model's decoder N times to serve as N first experts. Simultaneously, set a fully connected layer at the front end as the first router, resulting in a first hybrid expert module composed of N first experts and the router. The optimal value of N is set to 8. In this embodiment, after the FFN is replaced by the first hybrid expert module, the decoder structure is as follows: Figure 4 As shown.
[0064] Step 1.3: Sparsify the Cross-Attention modules (including Image-to-text Cross-Attention and Text-to-image Cross-Attention modules) in the Feature Enhancer module. Specifically, duplicate each of the four linear layers (q, k, v, and out) corresponding to each Cross-Attention module in the pre-trained model N times. Each of these N duplicated linear layers serves as N second experts. Simultaneously, a fully connected layer is set at the front end as a second router, thus replacing each linear layer with a sparsified second hybrid expert module. The second hybrid expert modules formed by the Query linear layer, Key linear layer, Value linear layer, and Out linear layer are denoted as Linear-Q MoE Layer, Linear-K MoE Layer, Linear-V MoE Layer, and Linear-OMoE Layer, respectively. After replacing the linear layers in each Cross-Attention module with the second hybrid expert module, it can be converted into a Cross-Attention MoE module (Image-to-text Cross-Attention MoE and Text-to-image Cross-Attention MoE). In this embodiment, after the linear layers are replaced with the second hybrid expert module, the Feature Enhancer structure is as follows: Figure 5 As shown.
[0065] Once the original Grounding DINO has completed the replacement of the first and second hybrid expert modules, it can be referred to as the hybrid expert DINO model.
[0066] Step 1.4: Set all first and second hybrid expert modules to learnable and freeze all other parameters.
[0067] Step 1.5: Set the training parameters: learning rate R = 0.0005, maximum number of iterations G = 120000.
[0068] Step 1.6: Fine-tune the DINO hybrid expert model based on training data consisting of image-text pairs (COCO, LVIS-minival, and LVIS-val can be used). During training, the input image-text pairs are processed by an image encoder and a text encoder to obtain image features and text features, respectively.
[0069] Step 1.7: Input the image features and text features into the multimodal encoder to obtain the enhanced image features and text features.
[0070] Step 1.8: Based on similarity calculation, select 900 image features that are most similar to the text features to initialize the query.
[0071] Step 1.9: Input 900 queries into the decoder to obtain the corresponding output.
[0072] Step 1.10: Input the data into the classification head and detection head to obtain the final predicted category and detection box.
[0073] Step 1.11: Use the sum of the loss function Loss of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules as the total loss function. Therefore, the actual total loss function used is Loss. all =Loss focal +Loss L1 +Loss giou +Loss aux Loss focal Loss L1 and Loss giou These represent the focal length loss, L1 loss, and GIOU loss originally used in the Grounding DINO model, respectively. aux This refers to the load balancing loss commonly used in MoE (Modal of Equilibrium). Since the Hybrid Expert DINO model contains a series of MoE modules, the load balancing loss needs to be calculated for each MoE module (including the first and second Hybrid Expert modules). The load balancing losses of all MoE modules are then summed to obtain the loss. aux .
[0074] Step 1.12: Based on the total loss value Loss all Calculate the gradient, backpropagate to update the learnable parameters of all MoE modules, and complete one round of fine-tuning training. Iterate this fine-tuning training until the maximum number of training iterations is reached.
[0075] To verify the effectiveness of the method proposed in this invention, tests were conducted on three datasets: COCO, LVIS-minival, and LVIS-val. The proposed method achieved zero-shot transfer AP scores of 48.4, 33.1, and 22.5 on the three datasets, respectively, representing improvements of 1.7, 4.3, and 2.4 compared to the Grounding DINO dataset. The overall results are shown in Tables 1, 2, and 3.
[0076] Table 1 Experimental results of the COCO dataset
[0077] Model AP Grounding DINO 48.4 MoE DINO(ours) 50.1
[0078] Table 2 Experimental results of the LVIS-minival dataset
[0079] Model AP Grounding DINO 28.8 MoE DINO(ours) 33.1
[0080] Table 3 Experimental results of the LVIS-val dataset
[0081] Model AP Grounding DINO 20.1 MoE DINO(ours) 22.5
[0082] Therefore, the method of this invention, which employs a hybrid expert fine-tuning approach, can improve the ability of the Grounding DINO model in open vocabulary object detection tasks.
[0083] The embodiments described above are merely some preferred implementations of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.
Claims
1. An open vocabulary target detection method based on hybrid expert fine-tuning, characterized in that, include: S1. Using the Grounding DINO model as a pre-trained model for open vocabulary target detection, the feedforward neural network module in the decoder is replaced with the first hybrid expert module. At the same time, each linear layer inside the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module is replaced with the second hybrid expert module, thus obtaining the hybrid expert DINO model. S2. Treat all parameters of the hybrid expert modules as learnable parameters, freeze all other parameters, use the sum of the loss function of the GroundingDINO model itself and the load balancing loss of all hybrid expert modules as the total loss, fine-tune the hybrid expert DINO model with training data composed of image-text pairs, and use the fine-tuned hybrid expert DINO model for open vocabulary target detection. The first hybrid expert module consists of multiple first experts and a first router. Each first expert is obtained by copying the original feedforward neural network module in the decoder. After receiving the module input, the first router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output. The second hybrid expert module consists of multiple second experts and a second router. Each second expert is obtained by copying the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the active expert according to the routing mechanism, and the outputs of each active expert are merged to generate the final module output. In the image-text cross-attention module and the text-image cross-attention module, each layer of query linear layer, key linear layer, value linear layer and output linear layer within the two modules needs to be replaced with an independent second hybrid expert module.
2. The open vocabulary target detection method based on hybrid expert fine-tuning as described in claim 1, characterized in that, The routing mechanism in the first router is a sparse routing mechanism.
3. The open vocabulary target detection method based on hybrid expert fine-tuning as described in claim 1, characterized in that, The routing mechanism in the second router is a sparse routing mechanism.
4. The open vocabulary target detection method based on hybrid expert fine-tuning as described in claim 1, characterized in that, The number of experts in both the first and second hybrid expert modules is set to 8.
5. An open vocabulary target detection system based on hybrid expert fine-tuning, characterized in that, include: The model building module is used to use the Grounding DINO model as a pre-trained model for open vocabulary object detection. It replaces the feedforward neural network module in the decoder with the first hybrid expert module, and replaces each linear layer inside the image-text cross attention module and the text-image cross attention module in the feature enhancement module with the second hybrid expert module, thereby obtaining the hybrid expert DINO model. The model fine-tuning module is used to treat all parameters of the hybrid expert modules as learnable parameters, freeze all other parameters, use the loss function of the Grounding DINO model itself and the sum of the load balancing loss of all hybrid expert modules as the total loss, fine-tune the hybrid expert DINO model with training data consisting of image-text pairs, and use the fine-tuned hybrid expert DINO model for open vocabulary target detection. The first hybrid expert module consists of multiple first experts and a first router. Each first expert is obtained by copying the original feedforward neural network module in the decoder. After receiving the module input, the first router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output. The second hybrid expert module consists of multiple second experts and a second router. Each second expert is obtained by copying the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the active expert according to the routing mechanism, and the outputs of each active expert are merged to generate the final module output. In the image-text cross-attention module and the text-image cross-attention module, each layer of query linear layer, key linear layer, value linear layer and output linear layer within the two modules needs to be replaced with an independent second hybrid expert module.
6. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the open vocabulary target detection method based on hybrid expert fine-tuning as described in any one of claims 1 to 4.
7. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the open vocabulary target detection method based on hybrid expert fine-tuning as described in any one of claims 1 to 4.