Cross-modal target detection method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and particularly discloses a cross-modal target detection method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting a to-be-detected image into an RAM model, and obtaining target text data; the target text data comprises a category name and a category probability respectively corresponding to each target in the to-be-detected image; and inputting the to-be-detected image and the target text data into a Grouping dino model for cross-modal feature fusion to obtain a target detection result of the to-be-detected image. According to the invention, through the adaptive modal fusion framework formed by the RAM model and the Grounding dino model, key information of target detection can be highlighted and noise information can be suppressed, so that the robustness and accuracy of target detection are improved.

Description

Background Art

[0002] With the explosive growth of the Internet and social media, the data generated and shared by people are becoming more and more diverse, including not only text, but also a large amount of visual content such as images and videos. This makes cross-modal fusion technology a hot topic in research.

[0003] Cross-modal learning aims to integrate information from different modalities and improve the ability to understand and process complex scenes through shared representation learning or joint modeling. However, there is heterogeneity between modalities. The essential differences in expression form and feature structure between text and visual modalities make it difficult to design an effective fusion mechanism. At the same time, data from different modalities often have uneven quality or contain noise. How to deal with incomplete or inconsistent cross-modal data and how to improve the robustness of the model are issues that need to be addressed urgently.

[0004] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a cross-modal target detection method, system, electronic device and storage medium.

[0006] In a first aspect, the present invention provides a cross-modal target detection method, and the technical solution of the method is as follows:

[0007] Input the image to be detected into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected;

[0008] The image to be detected and the target text data are input into the Grounding dino model for cross-modal feature fusion to obtain the target detection result of the image to be detected.

[0009] The beneficial effects of a cross-modal target detection method of the present invention are as follows:

[0010] The method of the present invention can highlight the key information of target detection and suppress noise information through the adaptive modal fusion framework formed by the RAM model and the Grounding dino model, thereby improving the robustness and accuracy of target detection.

[0011] Based on the above solution, a cross-modal target detection method of the present invention can also be improved as follows.

[0012] In an optional manner, the Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder;

[0013] The first Backbone module is used to perform feature extraction on the target text data to obtain a first text feature; the second Backbone module is used to perform feature extraction on the image to be detected to obtain a first image feature; the feature enhancer is used to perform feature extraction on the first text feature and the first image feature using a deep fusion method to obtain a second text feature and a second image feature; the query selection module is used to obtain a cross-modal query feature based on the second text feature and the second image feature; the cross-modal decoder is used to obtain a target detection result of the image to be detected based on the second text feature and the cross-modal query feature.

[0014] In an optional manner, the Grounding dino model further includes: a loss function module; the loss function module is used to iteratively train the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

[0015] In an optional manner, the method further includes:

[0016] According to the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when feature fusion of text features and image features is performed.

[0017] In a second aspect, the present invention provides a cross-modal target detection system, and the technical solution of the system is as follows:

[0018] It includes: a first detection unit and a second detection unit;

[0019] The first detection unit is used to: input the image to be detected into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected;

[0020] The second detection unit is used to: input the image to be detected and the target text data into a Grounding dino model for cross-modal feature fusion to obtain a target detection result of the image to be detected.

[0021] The beneficial effects of a cross-modal target detection system of the present invention are as follows:

[0022] The system of the present invention can highlight the key information of target detection and suppress noise information through the adaptive modal fusion framework formed by the RAM model and the Grounding dino model, thereby improving the robustness and accuracy of target detection.

[0023] Based on the above solution, a cross-modal target detection system of the present invention can also be improved as follows.

[0024] In an optional manner, the Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder;

[0025] The first Backbone module is used to perform feature extraction on the target text data to obtain a first text feature; the second Backbone module is used to perform feature extraction on the image to be detected to obtain a first image feature; the feature enhancer is used to perform feature extraction on the first text feature and the first image feature using a deep fusion method to obtain a second text feature and a second image feature; the query selection module is used to obtain a cross-modal query feature based on the second text feature and the second image feature; the cross-modal decoder is used to obtain a target detection result of the image to be detected based on the second text feature and the cross-modal query feature.

[0026] In an optional manner, the Grounding dino model further includes: a loss function module; the loss function module is used to iteratively train the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

[0027] In an optional manner, the method further includes: an allocating unit; the allocating unit is used to:

[0028] According to the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when feature fusion of text features and image features is performed.

[0029] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0030] The invention comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps of the cross-modal target detection method of the present invention are implemented.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0032] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the cross-modal target detection method of the present invention.

[0033] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0035] Figure 1 A schematic diagram of a flow chart of an embodiment of a cross-modal target detection method of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of the Grounding dino model;

[0037] Figure 3 This is a schematic diagram of the principle of Feature Enhancer;

[0038] Figure 4 is a schematic structural diagram of an embodiment of a cross-modal target detection system of the present invention;

[0039] Figure 5 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0041] Figure 1A flow chart of an embodiment of a cross-modal target detection method provided by the present invention is shown, and the cross-modal target detection method can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the cross-modal target detection method by calling computer-readable instructions stored in a memory through a processor. As Figure 1 As shown, the following steps are included:

[0042] S1. Input the image to be detected into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected.

[0043] Among them, the RAM model is short for Recognize Anything Model. The RAM model is a basic model for image labeling. The features of the RAM model include: ① No reliance on manual annotations: The RAM model uses large-scale image-text pairs for training, and obtains large-scale unlabeled image labels through automatic text semantic parsing, thereby avoiding the expensive cost of manual annotation. ② Strong zero-shot performance: The RAM model has demonstrated impressive zero-shot performance in multiple benchmarks and outperforms classic multimodal models such as CLIP and BLIP. ③ Open vocabulary recognition: The RAM model can recognize any category, not just the categories encountered in training, which greatly expands its practicality. ④ Efficient label reasoning: In the reasoning stage, the RAM model uses a lightweight image label recognition decoder to improve the efficiency of reasoning about image labels.

[0044] The overall architecture of the RAM model is similar to Tag2Text, and includes three key modules: ① Image encoder: used to extract features from images, using swing-Transformer as the image encoder. ② Image tag recognition decoder: used for labeling, which can be composed of 2 layers of Transformer. ③ Text encoder-decoder: can be composed of 12 layers of Transformer, used to generate text data.

[0045] Among them, the process of S1 is mainly to evaluate the quality of the text generated by the RAM model to ensure the quality of the modality. The RAM model can identify any common category of objects in the image, not limited to objects in a specific field, but widely covers various visual elements. The output results of the RAM model include the category name and category probability corresponding to each target in the image, and are presented in the form of text. The last layer of the Transformer of the RAM model is connected to a linear function, and the number of neurons output is equal to the number of output categories; finally, a sigmoid function is connected to get the probability of each category. The output category is obtained according to the set threshold. We use the probability of each category output by the sigmoid function as the quality evaluation standard of the text modality. The category name and category probability corresponding to each target are input into the Grounding dino model to participate in subsequent calculations to determine the reliability of each text data, so as to ensure that the impact of low-quality text modality data on the final decision is reduced.

[0046] S2. Input the image to be detected and the target text data into the Grounding dino model for cross-modal feature fusion to obtain the target detection result of the image to be detected.

[0047] Among them, the Grounding dino model converts the dino model into an open set object detector by integrating a text encoder into the dino model. This conversion enables the model to detect any object described in the input image, thus breaking through the limitation that the traditional target detection model can only recognize predefined category targets. The Grounding dino model includes: ① Open set object detection: The Grounding dino model is able to identify and detect objects beyond the predefined categories, which is achieved by integrating a text encoder and combining language and visual modalities. ② Feature enhancer: The model uses a feature enhancer to enhance the feature representation of the input data, which helps to improve the detection performance of the model. ③ Language-guided query selection: By calculating the similarity between text features and image features through dot product, the model is able to select the most relevant query, thereby more accurately locating the target object. ④ Cross-modal decoder: The decoder is able to fuse information from different modalities to generate accurate detection results. The target detection results include: detection information of each target contained in the image to be detected.

[0048] In an alternative approach, Figure 2 As shown, the Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder.

[0049] Among them, the first Backbone module is used to: perform feature extraction on the target text data to obtain a first text feature; the second Backbone module is used to: perform feature extraction on the image to be detected to obtain a first image feature; the feature enhancer is used to: use a deep fusion method to perform feature extraction on the first text feature and the first image feature to obtain a second text feature and a second image feature; the query selection module is used to: obtain a cross-modal query feature based on the second text feature and the second image feature; the cross-modal decoder is used to: obtain a target detection result of the image to be detected based on the second text feature and the cross-modal query feature.

[0050] In addition, the Grounding dino model also includes: a loss function module; the loss function module is used to iteratively train the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

[0051] exist Figure 2 In the model, Input Text is the target text data, Input Image is the image to be detected, TextBackbone is the first Backbone module, Image Backbone is the second Backbone module, Vanilla TextFeatures is the first text feature, Vanilla IMage Features is the second text feature, Feature Enhancer is the feature enhancer, Text Features is the second text feature, Image Features is the second image feature, Language-guide Query Selection is the query selection module, Cross-Modality Queries is the cross-modal query feature, Cross-Modality Decoder is the cross-modal decoder, ModelOutputs is the target detection result, and Contrastive loss is the loss function module.

[0052] In an optional manner, the method further includes:

[0053] According to the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when feature fusion of text features and image features is performed.

[0054] Among them, in this embodiment, an adaptive weight optimization algorithm is designed, which can dynamically adjust the weight allocation strategy of visual and text feature fusion (i.e., adjust the weight allocation of the first feature weight and the second feature weight) according to the timely feedback of the prediction results. Through gradient back propagation, the algorithm can learn the optimal weight adjustment strategy during the training process. The adaptive weight optimization algorithm is implemented in the modal fusion part (feature enhancer FeatureEnhancer) of the Grounding dino model.

[0055] The Grounding dino model uses the feature enhancer to achieve cross-modal feature fusion to align different modal features. This process is achieved through the deep fusion method. The specific process is as follows: Figure 3 The first text feature is extracted to obtain the second text feature, and the first image feature is extracted to obtain the second image feature.

[0056] It should be noted that the feature enhancer Feature Enhancer is implemented by the following formula:

[0057]

[0058] P i2t =X-MHA(O,P)=SoftMax(Attn T ) v W (out,L) ,i∈{0,1,...,L-1}

[0059] O i2t =X-MHA(O,P)=SoftMax(Attn)σP v W (out,I) ,i∈{0,1,...,L-1}

[0060] Among them, in the feature enhancer Feature Enhancer, DyHead is used as the image encoder and BERT is used as the text encoder. i represents the text features of the i-th stage, O 0 represents the initial text feature (token feature), P i represents the image features of the i-th stage, P 0 represents the initial image features (token features), X-MHA represents the cross-modal multi-head attention mode, L represents the number of DyHeadModules, and BERTLayer represents the additional layers added outside the pre-trained BERT model.

[0061] It should be noted that X-MHA is Figure 3 In the fusion part, the formula corresponding to X-MHA is:

[0062] O (q) =OW (q,I) ,P (q) =PW (q,L) ,P (v) =PW (v,L) ,O (v) =OW (v,I)

[0063]

[0064] O t2i =SoftMax(Attn)σP (v) W (out,I)

[0065] P i2t =SoftMax(Attn T ) (v) W (out,L)

[0066] Among them, {W (symbol,I) ,W (symbol,I) :symbol∈{q,v,out}} are trainable parameters, which are similar to the query layer, value layer and output linear layer in multi-head self-attention.

[0067] In the cross-modal multi-head attention model, each head computes the context vector of one modality by paying attention to the other modality. q and P q A similarity solution is performed, and then the text is weighted and summed to obtain visual features, and the visual is weighted and summed to obtain text features. The text and vision interact with each other. When performing the weighted sum with the text, an indicator σ is added to assign weights to the tokens obtained from each text. σ is the quality assessment score of the text information generated by the RAM model (the category name and category probability corresponding to the target). The value of σ is obtained through the sigmoid function of the RAM model. The sigmoid function converts the data into a weight value in the interval (0, 1). Text data with more noise obtains a lower weight, while data points with less noise obtain a higher weight. This will weaken the impact of noise data on the final result. This reduces the errors in the final result caused by the errors of the RAM model and improves the overall accuracy.

[0068] The technical solution of this embodiment can highlight the key information of target detection and suppress noise information through the adaptive modal fusion framework formed by the RAM model and the Grounding dino model, thereby improving the robustness and accuracy of target detection.

[0069] Figure 4 FIG. 2 is a schematic diagram showing a structure of an embodiment of a cross-modal target detection system 200 provided by the present invention. Figure 4 As shown, the system 200 includes: a first detection unit 210 and a second detection unit 220;

[0070] The first detection unit 210 is used to: input the image to be detected into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected;

[0071] The second detection unit 220 is used to: input the image to be detected and the target text data into the Grounding dino model for cross-modal feature fusion to obtain the target detection result of the image to be detected.

[0072] In an optional manner, the Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder;

[0073] The first Backbone module is used to perform feature extraction on the target text data to obtain a first text feature; the second Backbone module is used to perform feature extraction on the image to be detected to obtain a first image feature; the feature enhancer is used to perform feature extraction on the first text feature and the first image feature using a deep fusion method to obtain a second text feature and a second image feature; the query selection module is used to obtain a cross-modal query feature based on the second text feature and the second image feature; the cross-modal decoder is used to obtain a target detection result of the image to be detected based on the second text feature and the cross-modal query feature.

[0074] In an optional manner, the Grounding dino model further includes: a loss function module; the loss function module is used to iteratively train the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

[0075] In an optional manner, the method further includes: an allocating unit; the allocating unit is used to:

[0076] According to the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when feature fusion of text features and image features is performed.

[0077] It should be noted that the beneficial effects of the cross-modal target detection system provided by the above-mentioned embodiment are the same as the beneficial effects of the above-mentioned cross-modal target detection method, which will not be repeated here. In addition, when the system provided by the above-mentioned embodiment realizes its functions, it only takes the division of the above-mentioned functional units as an example. In practical applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the system is divided into different functional units according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0078] Among them, the cross-modal target detection system of the present invention can be a computer program (including program code) running in a computer device. For example, the cross-modal target detection system of the present invention is an application software that can be used to execute the corresponding steps in the cross-modal target detection method of the present invention.

[0079] In some embodiments, the cross-modal target detection system of the present invention can be implemented in a combination of software and hardware. As an example, the cross-modal target detection system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the cross-modal target detection method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0080] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0081] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned cross-modal target detection methods is implemented. That is, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the cross-modal target detection method shown in any embodiment of the present invention by calling the computer program.

[0082] In an alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0083] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0084] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0085] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0086] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0087] Among them, the electronic device can also be a terminal device, and the terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0088] It should be noted that Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0089] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned cross-modal target detection methods is implemented.

[0090] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0091] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the above-mentioned cross-modal target detection method.

[0092] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0093] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0094] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0095] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0096] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0097] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0098] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0099] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A cross-modal target detection method, characterized in that: include: Input the image to be detected into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected; The image to be detected and the target text data are input into the Grounding dino model for cross-modal feature fusion to obtain the target detection result of the image to be detected.

2. The cross-modal target detection method according to claim 1, characterized in that: The Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder; The first Backbone module is used to perform feature extraction on the target text data to obtain a first text feature; the second Backbone module is used to perform feature extraction on the image to be detected to obtain a first image feature; the feature enhancer is used to perform feature extraction on the first text feature and the first image feature using a deep fusion method to obtain a second text feature and a second image feature; the query selection module is used to obtain a cross-modal query feature based on the second text feature and the second image feature; the cross-modal decoder is used to obtain a target detection result of the image to be detected based on the second text feature and the cross-modal query feature.

3. The cross-modal target detection method according to claim 2, characterized in that: The Grounding dino model also includes: a loss function module; the loss function module is used to iteratively train the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

4. The cross-modal target detection method according to claim 2 or 3, characterized in that: Also includes: According to the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when feature fusion of text features and image features is performed.

5. A cross-modal target detection system, characterized in that: include: a first detection unit and a second detection unit; The first detection unit is used to: input the image to be detected into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected; The second detection unit is used to: input the image to be detected and the target text data into the Groundingdino model for cross-modal feature fusion to obtain the target detection result of the image to be detected.

6. The cross-modal target detection system according to claim 5, characterized in that: The Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder; The first Backbone module is used to perform feature extraction on the target text data to obtain a first text feature; the second Backbone module is used to perform feature extraction on the image to be detected to obtain a first image feature; the feature enhancer is used to perform feature extraction on the first text feature and the first image feature using a deep fusion method to obtain a second text feature and a second image feature; the query selection module is used to obtain a cross-modal query feature based on the second text feature and the second image feature; the cross-modal decoder is used to obtain a target detection result of the image to be detected based on the second text feature and the cross-modal query feature.

7. The cross-modal target detection system according to claim 6, characterized in that: The Grounding dino model also includes: a loss function module; the loss function module is used to iteratively train the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

8. The cross-modal target detection system according to claim 6 or 7, characterized in that: Also includes: The allocation unit is used for: According to the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when feature fusion of text features and image features is performed.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the cross-modal target detection method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the cross-modal target detection method according to any one of claims 1 to 4.

Images