Open vocabulary target detection method and system based on mixed expert fine tuning

Abstract

The invention discloses an open vocabulary target detection method and system based on mixed expert fine tuning. According to the invention, a Grounding DINO model is used as a pre-training model, and a hybrid expert (MoE) module is respectively introduced into a feedforward neural network module in an original decoder and a cross attention module in a feature enhancement module, so that different expert combinations can process tokens of different semantic information, thereby enhancing the open vocabulary detection capability of the network. According to the method, the hybrid expert system is introduced into the Grouping DINO model for the first time, and experimental results on a data set show that the open vocabulary detection capability of the model obtained through fine tuning is improved. Meanwhile, the method disclosed by the invention can also be used as a plug-in to be added into other Transform structures, and excessive reasoning overhead cannot be generated.

Description

Technical Field

[0001] The present invention relates to the field of open vocabulary object detection, and in particular to an open vocabulary object detection method and system based on hybrid expert fine-tuning. Background Art

[0002] In recent years, the field of open-vocabulary object detection has attracted widespread attention and gradually developed into a research hotspot. Unlike tasks that are trained on a specific set of categories, open-vocabulary object detection aims to detect objects in an image through text input.

[0003] Grounding DINO targets open-world multimodal object detection. It takes images and text as input and outputs the location of the target in the image indicated by the text. Through pre-training on massive data, it can achieve open-world zero-shot generalization.

[0004] In recent years, mixture of experts (MoE) models have been widely used in the fields of large language models (LLMs) and large multimodal models (MLLMs). Scaling law states that model size is a key factor in improving model performance. Given a limited computing resource budget, training a larger model with fewer training steps often yields better results than training a smaller model with more steps. A significant advantage of MoE is its ability to increase model size while maintaining inference speed, thereby enhancing performance. MoEs consist of two key components: sparse MoE layers, which replace the feed-forward network (FFN) layers in traditional Transformer models. MoE layers contain several experts, each of which is an independent neural network. The other component is a gating network or routing network, which directs input (tokens) to one or more designated expert networks. Token routing is a key aspect of MoE deployment. Typically, the routing network is a learnable parameter and is trained concurrently with the rest of the network.

[0005] However, there is currently no solution to how to introduce MoE into Grounding DINO to enable it to efficiently implement open vocabulary object detection. Summary of the Invention

[0006] The purpose of the present invention is to solve the technical problems existing in the prior art and to provide an open vocabulary object detection method and system based on hybrid expert fine-tuning.

[0007] The specific technical solutions adopted in the present invention are as follows:

[0008] In a first aspect, the present invention provides an open vocabulary object detection method based on hybrid expert fine-tuning, comprising:

[0009] S1. Use the Grounding DINO model as a pre-trained model for open vocabulary object detection. Replace the feedforward neural network module in the decoder with the first hybrid expert module. Also, replace each linear layer in the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module with the second hybrid expert module, thereby obtaining a hybrid expert DINO model.

[0010] S2. Parameters of all hybrid expert modules are set as learnable parameters, and all other parameters are frozen. The total loss is the sum of the loss function of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules. The hybrid expert DINO model is fine-tuned using training data consisting of image-text pairs, and the fine-tuned hybrid expert DINO model is used for open vocabulary object detection.

[0011] As a preferred embodiment of the above-mentioned first aspect, the first hybrid expert module is composed of multiple first experts and a first router, each first expert is obtained by copying the original feedforward neural network module in the decoder, and after receiving the module input, the first router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.

[0012] As a preferred embodiment of the first aspect, the routing mechanism in the first router adopts a sparse routing mechanism.

[0013] As a preferred embodiment of the above-mentioned first aspect, the second hybrid expert module is composed of multiple second experts and a second router. Each second expert is obtained by copying the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.

[0014] As a preferred embodiment of the first aspect, the routing mechanism in the second router adopts a sparse routing mechanism.

[0015] As a preferred embodiment of the above-mentioned first aspect, in the image-text cross-attention module and the text-image cross-attention module, each query (Query) linear layer, key (Key) linear layer, value (Value) linear layer, and output (Out) linear layer within the two modules need to be replaced with a mutually independent second hybrid expert module.

[0016] As a preference for the first aspect above, the number of experts in the first hybrid expert module and the second hybrid expert module is both set to 8.

[0017] In a second aspect, the present invention provides an open vocabulary object detection system based on hybrid expert fine-tuning, comprising:

[0018] A network construction module is used to use the Grounding DINO model as a pre-trained model for open vocabulary object detection, replace the feedforward neural network module in the decoder with the first hybrid expert module, and replace each linear layer in the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module with the second hybrid expert module, thereby obtaining the hybrid expert DINO model;

[0019] The network fine-tuning module is used to treat the parameters of all hybrid expert modules as learnable parameters and freeze all other parameters. The total loss is the sum of the loss function of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules. The hybrid expert DINO model is fine-tuned using training data consisting of image-text pairs, and the fine-tuned hybrid expert DINO model is used for open vocabulary object detection.

[0020] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the open vocabulary object detection method based on hybrid expert fine-tuning as described in any of the solutions in the first aspect above.

[0021] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0022] The memory is used to store computer programs;

[0023] The processor is configured to implement the open vocabulary object detection method based on hybrid expert fine-tuning as described in any one of the solutions of the first aspect when executing the computer program.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention proposes an open vocabulary target detection method based on hybrid expert fine-tuning. This method introduces a hybrid expert model on the basis of the traditional open vocabulary target detection algorithm to enhance the expressive ability of the model. The present invention transforms the decoder for the first time, converting the FFN module in the original decoder into a hybrid expert system composed of multiple FFNs. Different tokens can be processed by different expert combinations to improve the open vocabulary detection ability of the model. In order to accelerate training, the present invention adopts the Fine-tuning method. Based on the pre-trained Grounding DINO, the image text pair dataset is used to fine-tune the sparse MoE layer parameters and freeze all other parameters of the network. After experimental verification on the three datasets of COCO, LVIS-minival, and LVIS-val, the present invention can improve the zero-sample transfer AP index of the model on these three datasets, indicating that the open vocabulary detection ability of the model has been improved. At the same time, the method of the present invention can also be added as a plug-in to other Transformer structures without generating excessive inference overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the steps of the open vocabulary object detection method based on hybrid expert fine-tuning;

[0027] Figure 2 A diagram showing the module composition of an open vocabulary object detection system based on hybrid expert fine-tuning.

[0028] Figure 3 It is a schematic diagram of the structure of computer electronic equipment;

[0029] Figure 4 Schematic diagram of the structure after the first hybrid expert module is set in the decoder;

[0030] Figure 5 This is a structural diagram after setting the second hybrid expert module in the feature enhancement module. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0032] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0033] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for open vocabulary object detection based on hybrid expert fine-tuning is provided, which includes:

[0034] S1. Use the Grounding DINO model as the pre-trained model for open vocabulary object detection. Replace the feedforward neural network module (FFN) in the decoder layer with the first hybrid expert module. Meanwhile, replace each linear layer in the image-to-text cross-attention module and the text-to-image cross-attention module in the feature enhancement module with the second hybrid expert module, thus obtaining the hybrid expert DINO model.

[0035] S2. Parameters of all hybrid expert modules are treated as learnable parameters, and all other parameters are frozen. The total loss is the sum of the loss function of the Grounding DINO model itself and the load balancing loss (LBL) of all hybrid expert modules. The hybrid expert DINO model is fine-tuned using training data consisting of image-text pairs, and the fine-tuned hybrid expert DINO model is used for open vocabulary object detection.

[0036] It should be noted that the Grounding DINO model itself is state-of-the-art. Its network architecture consists of four key components: an image encoder (Image Backbone), a text encoder (Text Backbone), a feature enhancement module (Feature Enhancer), a language-guided query selection module (Language-guided Query Selection), and a cross-modality decoder (Cross-Modality Decoder). Specifically, Grounding DINO uses the SwinTransformer for the Image Backbone and BERT for the Text Backbone. The Feature Enhancer uses Deformable Self-Attention to enhance multi-scale image features and Self-Attention to enhance text features. Drawing on GLIP, image-to-text cross-attention and text-to-image cross-attention modules are added to help align features from different modalities. Finally, updated image / text features are obtained through a fully connected neural network. To guide text for object detection, a language-guided query selection mechanism is designed to select features more relevant to the text as queries for the decoder. Cross-Modality Decoder takes cross-modal query as input, uses Self-Attention to enhance features, and combines updated image and text features. It combines image features with Image Cross-Attention and text features with Text Cross-Attention. Finally, it uses a fully connected neural network to obtain the final features for generating detection boxes. L1 loss and GIOU loss are used for box regression. Referring to GLIP, contrast loss (specifically focal loss) is used to constrain whether the predicted target matches the text. Therefore, the loss function used for Grounding DINO model training is Loss = Loss focal +Loss L1 +Loss giou , Loss focal 、Loss L1 and Loss giouThey are focal loss, L1 loss, and GIOU loss, respectively. The specific network structure, loss function, and technical details of the Grounding DINO model can be found in the existing technical literature: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (https: / / arxiv.org / pdf / 2303.05499), which will not be repeated here.

[0037] It should be noted that the above-mentioned first hybrid expert module is composed of multiple first experts and a first router. Each first expert is obtained by copying the original feedforward neural network module in the decoder, that is, each first expert is an FFN. After receiving the module input, the first router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.

[0038] It should be noted that the routing mechanism in the first router can be selected according to actual needs. In theory, attention mechanism, soft routing, hard routing or sparse routing are all possible. In an embodiment of the present invention, the routing mechanism in the first router preferably adopts a sparse routing mechanism, which is a combination of soft routing and hard routing. It not only retains the advantages of dynamic allocation, but also improves efficiency by limiting the number of participating experts. In the sparse routing mechanism, a fully connected layer can be used as the first router. Assuming that the module input of the first hybrid expert module is X1, the module input X1 can be mapped to the routing probability of each first expert being activated through the fully connected layer. The k first experts with the highest routing probability are selected as activated experts in a Top-k manner. Each activated expert can calculate the expert output o according to the module input X1. i =FFN i (X1), FFN i () represents the i-th activated expert in the first hybrid expert module. The routing probabilities of all k activated experts are normalized by Softmax as weights, and then the outputs o of all k activated experts are normalized by Softmax. i The weighted summation is performed to generate the final module output o1 of the first hybrid expert module. This final module output o1 will replace the output of the original feedforward neural network module FFN in the decoder of Grounding DINO and be input into the subsequent cascaded Grounding DINO network layer.

[0039] Similarly, the second hybrid expert module is composed of multiple second experts and a second router. Each second expert is obtained by copying the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.

[0040] It should be noted that the routing mechanism in the second router can also be selected according to actual needs. In theory, attention mechanism, soft routing, hard routing or sparse routing can all be used. In an embodiment of the present invention, the routing mechanism in the second router also preferably adopts a sparse routing mechanism. In the sparse routing mechanism, a fully connected layer can be used as the second router. Assuming that the module input of the second hybrid expert module is X2, the module input X2 can be mapped to the routing probability of each second expert being activated through the fully connected layer. The k second experts with the highest routing probability are selected as activated experts in a Top-k manner. Each activated expert can calculate the expert output o according to the module input X2. i =Linear i (X2), Linear i () represents the i-th activated expert in the second hybrid expert module. The routing probabilities of all k activated experts are normalized by Softmax as weights, and then the outputs o of all k activated experts are normalized by Softmax. i The weighted summation is performed to generate the final module output o2 of the second hybrid expert module. This final module output o2 will replace the output of the original linear layer in the Grounding DINO decoder and be input into the subsequent cascaded Grounding DINO network layer.

[0041] It should be noted that in the image-text cross-attention module and the text-image cross-attention module, there are four linear layers q, k, v and out in both modules, which are respectively denoted as the query linear layer, the key linear layer, the value linear layer and the output linear layer. Each linear layer needs to be replaced by the second hybrid expert module respectively, and the second hybrid expert modules formed by replacing different linear layers are independent of each other.

[0042] It should also be noted that the number of experts in the first hybrid expert module and the second hybrid expert module can be optimized according to actual conditions. In the embodiment of the present invention, the number of experts in the first hybrid expert module and the second hybrid expert module is preferably set to 8.

[0043] This paper introduces a hybrid expert module to the existing FFN module in the decoder and the Cross-Attention module in the encoder, enabling different expert combinations to process tokens with different semantic information. This enhances the network's open vocabulary detection capabilities. This method is a fine-tuning method that modifies the decoder based on a pre-trained model and fine-tunes only the hybrid expert module in the modified decoder. This method can also be added as a plug-in to other Transformer architectures without incurring excessive inference overhead.

[0044] It should be noted that the method steps shown in S1 to S2 above can essentially be implemented in the form of computer programs or software function modules.

[0045] Therefore, based on the same inventive concept, Figure 2 As shown, the present invention also provides an open vocabulary object detection system based on hybrid expert fine-tuning corresponding to the open vocabulary object detection method based on hybrid expert fine-tuning provided in the above embodiment, which includes:

[0046] A network construction module is used to use the Grounding DINO model as a pre-trained model for open vocabulary object detection, replace the feedforward neural network module in the decoder with the first hybrid expert module, and replace each linear layer in the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module with the second hybrid expert module, thereby obtaining the hybrid expert DINO model;

[0047] The network fine-tuning module is used to treat the parameters of all hybrid expert modules as learnable parameters and freeze all other parameters. The total loss is the sum of the loss function of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules. The hybrid expert DINO model is fine-tuned using training data consisting of image-text pairs, and the fine-tuned hybrid expert DINO model is used for open vocabulary object detection.

[0048] In addition, based on the same inventive concept, Figure 3 As shown, the present invention also provides a computer electronic device corresponding to the open vocabulary object detection method based on hybrid expert fine-tuning provided in the above embodiment, which includes a memory and a processor;

[0049] The memory is used to store computer programs;

[0050] The processor is configured to implement the open vocabulary object detection method based on hybrid expert fine-tuning as described above when executing the computer program;

[0051] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0052] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to an open vocabulary target detection method based on hybrid expert fine-tuning, on which a computer program is stored. When the computer program is executed by a processor, the open vocabulary target detection method based on hybrid expert fine-tuning as described above can be implemented.

[0053] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the open vocabulary object detection method based on hybrid expert fine-tuning as described above.

[0054] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S2.

[0055] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0056] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0057] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0058] The present invention will further illustrate the detailed implementation process and technical effects of the open vocabulary object detection method based on hybrid expert model fine-tuning shown in steps S1 to S2 on a specific dataset through a specific embodiment, so as to facilitate understanding of the essence of the present invention.

[0059] Example

[0060] This embodiment refers to the open vocabulary object detection method based on hybrid expert model fine-tuning shown in the aforementioned steps S1 to S2, and demonstrates the model construction and fine-tuning process of this method on a specific dataset, while also demonstrating some specific parameter settings and implementation results.

[0061] In this embodiment, the open vocabulary object detection method based on fine-tuning of the hybrid expert model has the following specific steps:

[0062] Step 1.1: For the development of vocabulary object detection, use the Grounding DINO model as the pre-trained model and load the pre-trained weights.

[0063] Step 1.2: Sparsify the FFN module in the decoder of Grounding DINO. Specifically, copy the FFN model in the decoder of the pre-trained model N times as N first experts, and set a fully connected layer as the first router at the front end to obtain a first hybrid expert module consisting of N first experts and routers. The optimal value of N is set to 8. In this embodiment, after the FFN is replaced by the first hybrid expert module, the decoder structure is as follows: Figure 4 shown.

[0064] Step 1.3: Sparsify the Cross-Attention modules (including the Image-to-text Cross-Attention and Text-to-image Cross-Attention modules) in the Feature Enhancer. Specifically, replicate N linear layers (q, k, v, and out) corresponding to each Cross-Attention module in the pre-trained model. Each linear layer corresponds to N replicated linear layers as N second experts. A fully connected layer is set up at the front end as a second router, replacing each linear layer with a sparse second hybrid expert module. The second hybrid expert modules formed by the query linear layer, key linear layer, value linear layer, and output linear layer are denoted as Linear-Q MoE Layer, Linear-K MoE Layer, Linear-V MoE Layer, and Linear-O MoE Layer, respectively. After the linear layer in each Cross-Attention module is replaced by the second hybrid expert module, it can be converted into a Cross-Attention MoE module (Image-to-text Cross-Attention MoE and Text-to-image Cross-Attention MoE). In this embodiment, after the linear layer is replaced by the second hybrid expert module, the Feature Enhancer structure is as follows: Figure 5 shown.

[0065] When the original Grounding DINO completes the replacement of the first hybrid expert module and the second hybrid expert module, it can be recorded as a hybrid expert DINO model.

[0066] Step 1.4: Set all the first mixture of experts modules and the second mixture of experts modules to be learnable and freeze all other parameters.

[0067] Step 1.5: Set the training parameters, learning rate R = 0.0005, and maximum number of iterations G = 120,000.

[0068] Step 1.6: Fine-tune the hybrid expert DINO model based on training data consisting of image-text pairs (COCO, LVIS-minival, and LVIS-val can be used). During training, the image-text pair is input and the image and text features are obtained by passing them through the image encoder and text encoder, respectively.

[0069] Step 1.7: Input the image features and text features into the multimodal encoder to obtain enhanced image features and text features.

[0070] Step 1.8: Based on the similarity calculation, select 900 image features that are most similar to the text features to initialize the query.

[0071] Step 1.9: Input 900 queries into the decoder to get the corresponding output.

[0072] Step 1.10: Input into the classification head and detection head to obtain the final predicted category and detection box.

[0073] Step 1.11: The total loss function is the sum of the loss function of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules. The total loss function actually used is Loss all =Loss focal +Loss L1 +Loss giou +Loss aux , where Loss focal 、Loss L1 and Loss giou They are the focal loss, L1 loss and GIOU loss originally adopted by the Grounding DINO model, Loss aux is the load balancing loss commonly used in MoE. Since the hybrid expert DINO model has a series of MoE modules, each MoE module (including the first hybrid expert module and the second hybrid expert module) needs to calculate the load balancing loss, and then the load balancing losses of all MoE modules are summed as Loss. aux .

[0074] Step 1.12: According to the total loss value Loss all Calculate the gradient and back-propagate to update the learnable parameters of all MoE modules, completing a round of fine-tuning training. Continuously iterate the above fine-tuning training until the number of training iterations reaches the maximum number of iterations.

[0075] To validate the effectiveness of our method, we tested it on the COCO, LVIS-minival, and LVIS-val datasets. Our method achieved zero-shot transfer AP metrics of 48.4, 33.1, and 22.5 on these datasets, respectively, representing improvements of 1.7, 4.3, and 2.4 over Grounding DINO. The overall results are shown in Tables 1, 2, and 3.

[0076] Table 1 Experimental results of COCO dataset

[0077] Model AP Grounding DINO 48.4 MoE DINO(ours) 50.1

[0078] Table 2 Experimental results of LVIS-minival dataset

[0079] Model AP Grounding DINO 28.8 MoE DINO(ours) 33.1

[0080] Table 3 Experimental results of LVIS-val dataset

[0081] Model AP Grounding DINO 20.1 MoE DINO(ours) 22.5

[0082] It can be seen that the method of the present invention adopts the hybrid expert fine-tuning approach to improve the ability of the Grounding DINO model in the open vocabulary object detection task.

[0083] The embodiments described above are merely some preferred implementations of the present invention and are not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. An open vocabulary object detection method based on hybrid expert fine-tuning, characterized by: include: S1. Use the Grounding DINO model as a pre-trained model for open vocabulary object detection. Replace the feedforward neural network module in the decoder with the first hybrid expert module. Also, replace each linear layer in the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module with the second hybrid expert module, thereby obtaining a hybrid expert DINO model. S2. Parameters of all hybrid expert modules are set as learnable parameters, and all other parameters are frozen. The total loss is the sum of the loss function of the GroundingDINO model itself and the load balancing loss of all hybrid expert modules. The hybrid expert DINO model is fine-tuned using training data consisting of image-text pairs, and the fine-tuned hybrid expert DINO model is used for open vocabulary object detection.

2. The open vocabulary object detection method based on hybrid expert fine-tuning according to claim 1, characterized in that The first hybrid expert module is composed of multiple first experts and a first router. Each first expert is obtained by copying the original feedforward neural network module in the decoder. After receiving the module input, the first router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.

3. The open vocabulary object detection method based on hybrid expert fine-tuning according to claim 1, characterized in that The routing mechanism in the first router adopts a sparse routing mechanism.

4. The open vocabulary object detection method based on hybrid expert fine-tuning according to claim 1, characterized in that The second hybrid expert module is composed of multiple second experts and a second router. Each second expert is obtained by copying the linear layer replaced by the second hybrid expert module. After receiving the module input, the second router routes the module input to the activation expert according to the routing mechanism, and the outputs of each activation expert are fused to generate the final module output.

5. The open vocabulary object detection method based on hybrid expert fine-tuning according to claim 1, characterized in that The routing mechanism in the second router adopts a sparse routing mechanism.

6. The open vocabulary object detection method based on hybrid expert fine-tuning according to claim 1, characterized in that In the image-text cross-attention module and the text-image cross-attention module, each query (Query) linear layer, key (Key) linear layer, value (Value) linear layer, and output (Out) linear layer within the two modules need to be replaced with a second independent hybrid expert module.

7. The open vocabulary object detection method based on hybrid expert fine-tuning according to claim 1, characterized in that The number of experts in the first hybrid expert module and the second hybrid expert module is both set to 8.

8. An open vocabulary object detection system based on hybrid expert fine-tuning, characterized by: include: A model construction module is used to use the Grounding DINO model as a pre-trained model for open vocabulary object detection, replace the feedforward neural network module in the decoder with the first hybrid expert module, and replace each linear layer in the image-text cross-attention module and the text-image cross-attention module in the feature enhancement module with the second hybrid expert module, thereby obtaining the hybrid expert DINO model; A model fine-tuning module is used to treat the parameters of all hybrid expert modules as learnable parameters and freeze all other parameters. The total loss is the sum of the loss function of the Grounding DINO model itself and the load balancing loss of all hybrid expert modules. The hybrid expert DINO model is fine-tuned using training data consisting of image-text pairs, and the fine-tuned hybrid expert DINO model is used for open vocabulary object detection.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the open vocabulary object detection method based on hybrid expert fine-tuning according to any one of claims 1 to 7 can be implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the open vocabulary object detection method based on hybrid expert fine-tuning according to any one of claims 1 to 7 when executing the computer program.

Images