A cross-modal target detection method, system, electronic device and storage medium

Abstract

The application relates to the technical field of computer vision, and particularly discloses a cross-modal target detection method and system, an electronic device and a storage medium, the method comprising the following steps: inputting a to-be-detected image into a RAM model to obtain target text data; the target text data comprises a category name and a category probability corresponding to each target in the to-be-detected image; inputting the to-be-detected image and the target text data into a Grounding dino model for cross-modal feature fusion to obtain a target detection result of the to-be-detected image. Through the self-adaptive modal fusion framework formed by the RAM model and the Grounding dino model, key information of target detection can be highlighted and noise information can be suppressed, so that the robustness and accuracy of target detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a cross-modal target detection method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the explosive growth of the Internet and social media, the data generated and shared by people is becoming more and more diverse, not only containing text, but also a large amount of visual content such as images and videos. This makes cross-modal fusion technology a hot research topic.

[0003] Cross-modal learning aims to integrate information from different modalities, and improve the understanding and processing ability of complex scenes through shared representation learning or joint modeling. However, there is heterogeneity between modalities, and the essential differences in expression form and feature structure between text and visual modalities make it a big problem to design an effective fusion mechanism. At the same time, the data of different modalities often exist in the form of quality imbalance or contain noise, how to process incomplete or inconsistent cross-modal data, and how to improve the robustness of the model are current problems to be solved.

[0004] Therefore, there is an urgent need to provide a technical solution to solve the above problems. SUMMARY

[0005] To solve the above technical problems, the present application provides a cross-modal target detection method, system, electronic device and storage medium.

[0006] In a first aspect, the present application provides a cross-modal target detection method, and the technical scheme of the method is as follows:

[0007] inputting the to-be-detected image into the RAM model to obtain target text data; the target text data contains a class name and a class probability corresponding to each target in the to-be-detected image respectively;

[0008] inputting the to-be-detected image and the target text data into the Grounding dino model for cross-modal feature fusion to obtain a target detection result of the to-be-detected image.

[0009] The cross-modal target detection method of the present application has the following beneficial effects:

[0010] The method of the present application can highlight the key information of target detection and suppress noise information through the self-adaptive modal fusion framework formed by the RAM model and the Grounding dino model, thereby improving the robustness and accuracy of target detection.

[0011] On the basis of the above-mentioned scheme, the cross-modal target detection method of the present application can also be improved as follows.

[0012] In an optional mode, the Grounding dino model comprises a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder.

[0013] The first Backbone module is configured to perform feature extraction on the target text data to obtain first text features; the second Backbone module is configured to perform feature extraction on the to-be-detected image to obtain first image features; the feature enhancer is configured to perform feature extraction on the first text features and the first image features by using a deep fusion mode to obtain second text features and second image features; the query selection module is configured to obtain cross-modal query features according to the second text features and the second image features; and the cross-modal decoder is configured to obtain a target detection result of the to-be-detected image according to the second text features and the cross-modal query features.

[0014] In an optional mode, the Grounding dino model further comprises a loss function module; and the loss function module is configured to iteratively train the Grounding dino model according to label data of the to-be-detected image and a prediction result of the sample image.

[0015] In an optional mode, the system further comprises:

[0016] According to the class name and the class probability corresponding to each target in the to-be-detected image, a first feature weight corresponding to the target text data and a second feature weight corresponding to the to-be-detected image are determined; and the first feature weight and the second feature weight are used for weight distribution when text features and image features are fused.

[0017] In a second aspect, the present application provides a cross-modal target detection system, and the technical scheme of the system is as follows:

[0018] The system comprises a first detection unit and a second detection unit.

[0019] The first detection unit is configured to input a to-be-detected image into a RAM model to obtain target text data; and the target text data comprises a class name and a class probability corresponding to each target in the to-be-detected image.

[0020] The second detection unit is configured to input the to-be-detected image and the target text data into a Grounding dino model to perform cross-modal feature fusion and obtain a target detection result of the to-be-detected image.

[0021] The cross-modal target detection system of the present application has the following advantages:

[0022] The adaptive modal fusion framework formed by the RAM model and the Grounding dino model can highlight the key information of target detection and suppress noise information, thereby improving the robustness and accuracy of target detection.

[0023] On the basis of the above scheme, the cross-modal target detection system of the present application can be further improved as follows.

[0024] In an optional manner, the Grounding dino model comprises a first Backbone module, a second Backbone module, a feature enhancer, a query selection module and a cross-modal decoder.

[0025] The first Backbone module is configured to perform feature extraction on the target text data to obtain first text features; the second Backbone module is configured to perform feature extraction on the image to be detected to obtain first image features; the feature enhancer is configured to perform feature extraction on the first text features and the first image features by using a deep fusion method to obtain second text features and second image features; the query selection module is configured to obtain cross-modal query features according to the second text features and the second image features; and the cross-modal decoder is configured to obtain the target detection result of the image to be detected according to the second text features and the cross-modal query features.

[0026] In an optional manner, the Grounding dino model further comprises a loss function module; the loss function module is configured to perform iterative training on the Grounding dino model according to the label data of the image to be detected and the prediction result of the sample image.

[0027] In an optional manner, it further comprises an allocation unit; the allocation unit is configured to:

[0028] According to the category name and category probability corresponding to each target in the image to be detected, determine the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected; the first feature weight and the second feature weight are used for weight allocation when text features and image features are fused.

[0029] In a third aspect, the technical scheme of an electronic device of the present application is as follows:

[0030] comprising a memory, a processor and a program stored on the memory and running on the processor, the processor executing the program to realize the steps of the cross-modal target detection method of the present application.

[0031] In a fourth aspect, the present application provides a computer readable storage medium comprising instructions, which when read by a computer, cause the computer to perform the steps of the cross-modal object detection method of the present application.

[0032] The computer readable storage medium stores instructions, which when read by the computer readable storage medium, cause the computer readable storage medium to perform the steps of the cross-modal object detection method of the present application.

[0033] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application and to implement the same according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of the detailed description. In the drawings:

[0035] Figure 1 FIG. 1 is a flowchart of an embodiment of the cross-modal object detection method of the present application;

[0036] Figure 2 FIG. 2 is a structural diagram of the Grounding dino model;

[0037] Figure 3 FIG. 3 is a schematic diagram of the principle of the Feature Enhancer;

[0038] Figure 4 FIG. 4 is a structural diagram of an embodiment of the cross-modal object detection system of the present application;

[0039] Figure 5 FIG. 5 is a structural diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein.

[0041] Figure 1A flowchart of an embodiment of a cross-modal object detection method provided by the present application is shown, which can be executed by electronic devices such as terminal devices or servers. Among them, the terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the cross-modal object detection method by calling the computer readable instructions stored in the memory through the processor. As shown in Figure 1 includes the following steps:

[0042] S1, input the image to be detected into the RAM model to obtain target text data; the target text data contains the class name and class probability corresponding to each target in the image to be detected.

[0043] Among them, RAM model is the abbreviation of Recognize Anything Model, and the RAM model is an image labeling basic model. The characteristics of the RAM model include: ①Independent of manual annotation: the RAM model is trained using large-scale image-text pairs, and large-scale unlabeled image labels are obtained through automatic text semantic analysis, thereby avoiding the high cost of manual annotation. ②Strong zero-shot performance: the RAM model shows impressive zero-shot performance in multiple benchmark tests and outperforms classic multi-modal models such as CLIP and BLIP. ③Open vocabulary recognition: the RAM model can recognize any class, not just the classes encountered during training, thereby greatly expanding its practicality. ④Efficient label inference: in the inference stage, the RAM model uses a lightweight image label recognition decoder to improve the inference efficiency of image labels.

[0044] Among them, the overall architecture of the RAM model is similar to Tag2Text, which includes three key modules: ①Image encoder: used for feature extraction of images, using swing-Transformer as the image encoder. ②Image label recognition decoder: used for labeling, which can be composed of 2 layers of Transformer. ③Text encoder-decoder: can be composed of 12 layers of Transformer, used for generating text data.

[0045] The S1 process primarily evaluates the quality of the text generated by the RAM model, thereby ensuring the quality of the modality. The RAM model can recognize any common category of objects in an image, not limited to objects from specific domains, but broadly covering various visual elements. The output of the RAM model includes the category name and probability corresponding to each object in the image, presented in text form. The last Transformer layer of the RAM model is connected to a linear function, with the number of output neurons equal to the number of output categories; finally, a sigmoid function is connected to obtain the probability of each category. The output category is determined based on a set threshold. We use the probability of each category output by the sigmoid function as the quality evaluation standard for the text modality. The category name and probability corresponding to each object are input into the Grounding Dino model for subsequent calculations to determine the reliability of each text data point, thus ensuring that low-quality text modality data has a reduced impact on the final decision.

[0046] S2. Input the image to be detected and the target text data into the Grounding dino model to perform cross-modal feature fusion to obtain the target detection result of the image to be detected.

[0047] The Grounding Dino model integrates a text encoder into the Dino model, transforming it into an open-set object detector. This transformation enables the model to detect any object described in the input image, overcoming the limitation of traditional object detection models that can only recognize predefined categories. The Grounding Dino model includes: ① Open-set object detection: The Grounding Dino model can identify and detect objects beyond predefined categories, achieved by integrating a text encoder and combining linguistic and visual modalities. ② Feature enhancer: The model uses a feature enhancer to enhance the feature representation of the input data, which helps improve the model's detection performance. ③ Language-guided query selection: By calculating the similarity between text features and image features through dot product, the model can select the most relevant query, thereby more accurately locating the target object. ④ Cross-modal decoder: The decoder can fuse information from different modalities to generate accurate detection results. The object detection results include: detection information for each target contained in the image to be detected.

[0048] In one alternative approach, such as Figure 2 As shown, the Grounding Dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module, and a cross-modal decoder.

[0049] The first Backbone module is configured to perform feature extraction on the target text data to obtain first text features; the second Backbone module is configured to perform feature extraction on the to-be-detected image to obtain first image features; the feature enhancer is configured to perform feature extraction on the first text features and the first image features by using a deep fusion manner to obtain second text features and second image features; the query selection module is configured to obtain cross-modal query features according to the second text features and the second image features; and the cross-modal decoder is configured to obtain a target detection result of the to-be-detected image according to the second text features and the cross-modal query features.

[0050] In addition, the Grounding dino model further includes a loss function module, which is configured to perform iterative training on the Grounding dino model according to label data of the to-be-detected image and a prediction result of the sample image.

[0051] In Figure 2 , Input Text is target text data, Input Image is a to-be-detected image, TextBackbone is a first Backbone module, Image Backbone is a second Backbone module, Vanilla TextFeatures is first text features, Vanilla IMage Features is second text features, Feature Enhancer is a feature enhancer, Text Features is second text features, Image Features is second image features, Language-guide Query Selection (language-guided query selection) is a query selection module, Cross-Modality Queries is cross-modal query features, Cross-Modality Decoder is a cross-modal decoder, ModelOutputs is a target detection result, and Contrastive loss is a loss function module.

[0052] In an optional manner, the method further includes:

[0053] According to the category name and the category probability corresponding to each target in the to-be-detected image, a first feature weight corresponding to the target text data and a second feature weight corresponding to the to-be-detected image are determined; the first feature weight and the second feature weight are used for weight distribution when text features and image features are fused.

[0054] In the embodiment, an adaptive weight optimization algorithm is designed, which can dynamically adjust the weight distribution strategy of visual and text feature fusion (i.e., adjust the weight distribution of the first feature weight and the second feature weight) according to the timely feedback of the prediction result. Through gradient back propagation, the algorithm can learn the optimal weight adjustment strategy during the training process. The adaptive weight optimization algorithm is implemented in the modal fusion part (Feature Enhancer) of the Grounding dino model.

[0055] The Grounding dino model realizes cross-modal feature fusion through the Feature Enhancer to align different modal features. This process is realized through the deep fusion method, and the specific process is as shown in Figure 3 The first text feature is extracted to obtain the second text feature, and the first image feature is extracted to obtain the second image feature.

[0056] It should be noted that the Feature Enhancer is realized by the following formula:

[0057]

[0058] P i2t =X-MHA(O,P)=SoftMax(Attn T )O v W (out,L) ,i∈{0,1,...,L-1}

[0059] O t2i =X-MHA(O,P)=SoftMax(Attn)σP v W (out,I) ,i∈{0,1,...,L-1}

[0060] In the Feature Enhancer, DyHead is used as the image encoder, BERT is used as the text encoder, O i represents the text feature of the i-th stage, O 0 represents the initial text feature (token feature), P i represents the image feature of the i-th stage, P 0 represents the initial image feature (token feature), X-MHA represents the cross-modal multi-head attention mode, l represents the number of DyHeadModule, and BERTLayer represents the layer additionally added to the pre-trained BERT model.

[0061] It should be noted that X-MHA isFigure 3 where fusion is the fusion part in the formula of X-MHA:

[0062] O (q) = OW (q,I) ,P (q) = PW (q,L) ,P (v) = PW (v,L) ,O (v) = OW (v,I)

[0063]

[0064] O t2i = SoftMax(Attn)σP (v) W (out,I)

[0065] P i2t = SoftMax(Attn T )O (v) W (out,L)

[0066] where {W (symbol,I) ,W (symbol,I) :symbol∈{q,v,out}} are trainable parameters, which are similar to the query layer, value layer and output linear layer in multi-head self-attention.

[0067] In the cross-modal multi-head attention mode, each head calculates the context vector of one mode by focusing on the other mode. First, the similarity of O q and P q is solved, and then the visual features are weighted and summed to obtain the text features, and the visual features are weighted and summed to obtain the text features. The text and the visual have an interaction. In the weighted sum with the text, an index σ is added to assign weights to each token obtained by the text. σ is the quality evaluation score of the text information generated by the RAM model (the category name corresponding to the target and the category probability). The value of σ is obtained by the sigmoid function of the RAM model, which converts the data into a weight value in the interval (0, 1) by the sigmoid function. The text data with high noise obtains a low weight, and the data point with low noise obtains a high weight. In this way, the influence of noise data on the final result is weakened. Thus, the errors of the RAM model leading to the errors of the final result are reduced, and the overall accuracy is improved.

[0068] The technical scheme of the embodiment can highlight the key information of target detection and suppress noise information through the adaptive modal fusion framework formed by the RAM model and the Grounding dino model, thereby improving the robustness and accuracy of target detection.

[0069] Figure 4 A structural diagram of an embodiment of a cross-modal object detection system 200 provided by the present application is shown. As shown, the system 200 includes a first detection unit 210 and a second detection unit 220. Figure 4 The first detection unit 210 is configured to input a to-be-detected image into a RAM model to obtain target text data, wherein the target text data includes a class name and a class probability corresponding to each target in the to-be-detected image.

[0070] The second detection unit 220 is configured to input the to-be-detected image and the target text data into a Grounding dino model for cross-modal feature fusion to obtain an object detection result of the to-be-detected image.

[0071] The second detection unit 220 is configured to input the to-be-detected image and the target text data into a Grounding dino model for cross-modal feature fusion to obtain an object detection result of the to-be-detected image.

[0072] In an optional manner, the Grounding dino model includes a first Backbone module, a second Backbone module, a feature enhancer, a query selection module, and a cross-modal decoder.

[0073] The first Backbone module is configured to perform feature extraction on the target text data to obtain first text features; the second Backbone module is configured to perform feature extraction on the to-be-detected image to obtain first image features; the feature enhancer is configured to perform feature extraction on the first text features and the first image features by using a deep fusion manner to obtain second text features and second image features; the query selection module is configured to obtain cross-modal query features according to the second text features and the second image features; and the cross-modal decoder is configured to obtain the object detection result of the to-be-detected image according to the second text features and the cross-modal query features.

[0074] In an optional manner, the Grounding dino model further includes a loss function module, and the loss function module is configured to perform iterative training on the Grounding dino model according to label data of a to-be-detected image and a prediction result of a sample image.

[0075] In an optional manner, the system further includes an allocation unit, and the allocation unit is configured to:

[0076] According to the category name and the category probability corresponding to each target in the to-be-detected image, a first feature weight corresponding to the target text data and a second feature weight corresponding to the to-be-detected image are determined; the first feature weight and the second feature weight are used for weight distribution when text features and image features are fused.

[0077] It should be noted that the beneficial effects of the cross-modal target detection system provided by the above embodiments are the same as those of the cross-modal target detection method, which will not be repeated here. In addition, when the system provided by the above embodiments implements its functions, only the division of the above functional units is exemplified, and in actual application, the above functions can be completed by different functional units according to needs, that is, the system is divided into different functional units according to actual conditions to complete all or part of the above described functions. In addition, the system and method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0078] Among them, the cross-modal target detection system of the application can be a computer program (including program code) running in a computer device, for example, the cross-modal target detection system of the application is an application software, which can be used to execute the corresponding steps in the cross-modal target detection method of the application.

[0079] In some embodiments, the cross-modal target detection system of the application can be realized in a combination of software and hardware, for example, the cross-modal target detection system of the application can be a hardware decoding processor form of processor, which is programmed to execute the cross-modal target detection method of the application, for example, the hardware decoding processor form of processor can use one or more application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate Array) or other electronic components.

[0080] Among them, the modules involved in the embodiments of the application can be realized by software or hardware. Among them, the name of the module does not constitute a limitation of the module itself in some cases.

[0081] The electronic device of the embodiment of the present application can include, but is not limited to, a processor and a memory; the memory is configured to store a computer program; and the processor is configured to execute the cross-modal object detection method shown in any of the embodiments of the present application by calling the computer program.

[0082] In an optional embodiment, an electronic device is provided, as shown in Figure 5 Figure 5 The electronic device 4000 shown in the embodiment of the present application includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, through a bus 4002. Optionally, the electronic device 4000 can further include a transceiver 4004, which can be used for data interaction, such as data transmission and / or data reception, between the electronic device and other electronic devices. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0083] The processor 4001 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0084] The bus 4002 can include a path for transmitting information between the above-mentioned components. The bus 4002 can be a PCI (Peripheral Component Interconnect, peripheral component interconnect) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of expression, Figure 5 ​Only one bus 4002 is shown, but it could be comprised of several buses. Bus 4002 is used to transmit and receive electrical, electromagnetic or optical signals that include information to be sent and received by the various components of the computer 4000.

[0085] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions; a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions; an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0086] The memory 4003 is used to store application program codes (computer programs) for implementing the solutions of the present application, and is controlled by the processor 4001 to execute. The processor 4001 is used to execute the application program codes stored in the memory 4003 to realize the contents shown in the foregoing method embodiments.

[0087] The electronic device can also be a terminal device, which can be any terminal device that can install an application and access a webpage through the application, including at least one of a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, and a smart vehicle device.

[0088] It should be noted that, Figure 5 The electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0089] The computer readable storage medium of the embodiments of the present application, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize any one of the above cross-modal target detection methods.

[0090] Optionally, the computer readable storage medium can be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0091] In an example embodiment, a computer program product or computer program including computer instructions stored in a computer readable storage medium is also provided. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the above cross-modal object detection method.

[0092] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0093] It should be understood that the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of various embodiments of the present application. In this regard, each block in the flowchart and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the flowchart or block diagrams can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0094] The computer readable storage medium provided by the embodiments of the present application can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus or device.

[0095] The computer readable storage medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the embodiments described above.

[0096] The above description is merely the preferred embodiments of the present application and the explanation of the technical principles used. It should be understood by those skilled in the art that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present application (but not limited to) having similar functions.

[0097] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and represent no specific order or sequential order. The order of use of similar objects can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0098] Those skilled in the art know that the present application can be implemented as a system, a method or a computer program product, so the present application can be specifically implemented as follows: it can be a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" in this paper. In addition, in some embodiments, the present application can also be implemented as a computer program product in one or more computer readable media, which contains computer readable program code.

[0099] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A cross-modal target detection method, characterized in that, include: The image to be detected is input into the RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected. The image to be detected and the target text data are input into the Grounding Dino model for cross-modal feature fusion to obtain the target detection result of the image to be detected; The Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module, and a cross-modal decoder; The first Backbone module is used to: extract features from the target text data to obtain first text features; the second Backbone module is used to: extract features from the image to be detected to obtain first image features; the feature enhancer is used to: extract features from the first text features and the first image features using deep fusion to obtain second text features and second image features; the query selection module is used to: obtain cross-modal query features based on the second text features and the second image features; the cross-modal decoder is used to: obtain the target detection result of the image to be detected based on the second text features and the cross-modal query features. The feature enhancer is implemented using the following formula: P i2t = X - MHA(O, P) = SoftMax(Attn T )O v W (out,L) , i e {0, 1,..., L - 1} O t2i = X-MHA(O, P) = SoftMax(Attn)σP v W (out,I) i e {0, 1,..., L - 1} Wherein, in the feature enhancer, DyHead is used as an image encoder, BERT is used as a text encoder, O i represents the text feature of the i-th stage, O 0 represents the initial text feature, P i represents the image feature of the i-th stage, P 0 represents the initial image feature, X-MHA represents a cross-modal multi-head attention mode, L represents the number of DyHeadModule, BERTLayer represents a layer additionally added outside the pre-trained BERT model; σ is a quality evaluation score of the text information generated by the RAM model, and the quality evaluation score is: the category probability corresponding to the target containing the category name; the value of σ is obtained through the sigmoid function of the RAM model, the data is converted into a weight value in the interval (0, 1) through the sigmoid function, the weight obtained by the text data with larger noise is lower, and the data point with smaller noise obtains higher weight; The formula corresponding to X-MHA is: O (q) = OW (q,I) ,P (q) = PW (q,L) ,P (v) = PW (v,L) ,O (v) = OW (v,I) O t2i =SoftMax(Attn)σP (v) W (out,I) P i2t =SoftMax(Attn T )O (v) W (out,L) Among them, {W (symbol,I) W (symbol,I) :symbol∈{q,v,out}} are trainable parameters; It also includes: determining the first feature weight corresponding to the target text data and the second feature weight corresponding to the target image based on the category name and category probability corresponding to each target in the image to be detected; the first feature weight and the second feature weight are used for weight allocation when performing feature fusion of text features and image features.

2. The cross-modal target detection method according to claim 1, characterized in that, The Grounding dino model further includes a loss function module; the loss function module is used to iteratively train the Grounding dino model based on the label data of the image to be detected and the prediction results of the sample images.

3. A cross-modal target detection system, characterized in that, include: First detection unit and second detection unit; The first detection unit is used to: input the image to be detected into a RAM model to obtain target text data; the target text data includes the category name and category probability corresponding to each target in the image to be detected; The second detection unit is used to: input the image to be detected and the target text data into the Groundingdino model to perform cross-modal feature fusion, and obtain the target detection result of the image to be detected; The Grounding dino model includes: a first Backbone module, a second Backbone module, a feature enhancer, a query selection module, and a cross-modal decoder; The first Backbone module is used to: extract features from the target text data to obtain first text features; the second Backbone module is used to: extract features from the image to be detected to obtain first image features; the feature enhancer is used to: extract features from the first text features and the first image features using deep fusion to obtain second text features and second image features; the query selection module is used to: obtain cross-modal query features based on the second text features and the second image features; the cross-modal decoder is used to: obtain the target detection result of the image to be detected based on the second text features and the cross-modal query features. The feature enhancer is implemented using the following formula: P i2t =X-MHA(O,P)=SoftMax(Attn T )O v W (out,L) ,i∈{0,1,...,L-1} O t2i =X-MHA(O,P)=SoftMax(Attn)σP V W (out,I) ,i∈{0,1,...,L-1} In the feature enhancer, DyHead is used as the image encoder, and BERT is used as the text encoder. i O represents the text features of the i-th stage. 0 P represents the initial text features. i P represents the image features of the i-th stage. 0 σ represents the initial image features, X-MHA represents the cross-modal multi-head attention modality, L represents the number of DyHeadModules, and BERTLayer represents the additional layer added outside the pre-trained BERT model; σ is the quality assessment score of the text information generated by the RAM model, which is the class probability corresponding to the target containing the class name; the value of σ is obtained through the sigmoid function of the RAM model. The sigmoid function converts the data into weight values ​​in the (0,1) interval. Noisier text data receives lower weights, while noisier data points receive higher weights. The formula corresponding to X-MHA is: O (q) =OW (q,I) ,P (q) =PW (q,L) ,P (v) =PW (v,L) ,O (v) =OW (v,I) O t2i =SoftMax(Attn)σP (v) W (out,I) P i2t =SoftMax(Attn T )O (v) W (out,L) Among them, {W (symbol,I) W (symbol,I) :symbol∈{q,v,out}} are trainable parameters; It also includes: an allocation unit; the allocation unit is used for: Based on the category name and category probability corresponding to each target in the image to be detected, the first feature weight corresponding to the target text data and the second feature weight corresponding to the image to be detected are determined; the first feature weight and the second feature weight are used for weight allocation when the text features and image features are fused.

4. The cross-modal target detection system according to claim 3, characterized in that, The Grounding dino model further includes a loss function module; the loss function module is used to iteratively train the Grounding dino model based on the label data of the image to be detected and the prediction results of the sample images.

5. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the cross-modal target detection method as described in claim 1 or 2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer-readable storage medium to implement the cross-modal target detection method as described in claim 1 or 2.

Images