Target detection method and device and storage medium

Abstract

The invention discloses a target detection method and device, and a storage medium. The method comprises the following steps: determining a target detection network; the target detection network comprises a text encoder, an image encoder, a cross-modal cross-scale fusion module and a detection head; receiving image data and text information; in a text encoder, encoding the text information into a first semantic feature; in an image encoder, encoding the image data into image features of a plurality of scales; in a cross-modal cross-scale fusion module, the first semantic feature and the image features of multiple scales are locally and interactively fused into a multi-modal semantic feature from local to global; and in the detection head, generating a detection box of which the content is text information in the image data according to the multi-modal semantic features. Part of image features are used in the early stage, all image features are used in the later stage, under the condition that the image features are not lost, the target detection efficiency based on open vocabularies can be greatly improved, and the method is suitable for being deployed in an edge box with limited computing resources.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to a target detection method, device and storage medium. Background Art

[0002] In business scenarios such as video surveillance of rail transit, one or more video streams are collected, and users enter open vocabulary based on the needs of the business scenario, and real-time target detection is performed in the local video stream. For example, when the user enters the open vocabulary "fallen person", target detection is performed in the video stream to monitor the fallen person.

[0003] At present, the object detection method based on open vocabulary mainly adopts the structural design of dual encoding fusion, which uses all features in encoding fusion and consumes a lot of computing resources. The efficiency of object detection based on open vocabulary is low and there are limitations in edge boxes with less resources. Summary of the invention

[0004] In view of this, the present invention provides a target detection method, device and storage medium to improve the efficiency of open vocabulary-based target detection.

[0005] A first aspect of the present invention provides a target detection method, comprising:

[0006] Determine a target detection network; the target detection network includes a text encoder, an image encoder, a cross-modal and cross-scale fusion module and a detection head;

[0007] receiving image data and text information;

[0008] In the text encoder, the text information is encoded into a first semantic feature;

[0009] In the image encoder, encoding the image data into image features at multiple scales;

[0010] In the cross-modal and cross-scale fusion module, the first semantic feature and the image features of multiple scales are locally and globally interactively fused into a multi-modal semantic feature;

[0011] In the detection head, a detection frame containing the text information is generated in the image data according to the multimodal semantic features.

[0012] A second aspect of the present invention provides a target detection device, comprising:

[0013] A network determination module, used to determine a target detection network; the target detection network includes a text encoder, an image encoder, a cross-modal and cross-scale fusion module and a detection head;

[0014] A detection information receiving module, used for receiving image data and text information;

[0015] A text encoding module, used for encoding the text information into a first semantic feature in the text encoder;

[0016] An image encoding module, used for encoding the image data into image features of multiple scales in the image encoder;

[0017] An image-text interactive fusion module, configured to interactively fuse the first semantic feature with the image features of multiple scales locally from local to global into a multimodal semantic feature in the cross-modal and cross-scale fusion module;

[0018] A detection frame output module is used to generate a detection frame whose content is the text information in the image data according to the multimodal semantic features in the detection head.

[0019] A third aspect of the present invention provides an electronic device, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the target detection method as described in the first aspect above.

[0023] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target detection method as described in the first aspect above is implemented.

[0024] A fifth aspect of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the target detection method as described in the first aspect above.

[0025] In this embodiment, a target detection network is determined; the target detection network includes a text encoder, an image encoder, a cross-modal and cross-scale fusion module and a detection head; image data and text information are received; in the text encoder, the text information is encoded into a first semantic feature; in the image encoder, the image data is encoded into image features of multiple scales; in the cross-modal and cross-scale fusion module, the first semantic feature and the image features of multiple scales are locally transformed from local to global interactive fusion into a multi-modal semantic feature; in the detection head, a detection frame with text information is generated in the image data based on the multi-modal semantic feature. Using some image features in the early stage and all image features in the later stage can greatly improve the efficiency of open vocabulary-based target detection without losing image features, and is suitable for deployment in edge boxes with limited computing resources.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 This is a flow chart of a target detection method provided in Example 1 of the present invention.

[0029] Figure 2 It is a schematic diagram of a target detection network provided in Example 1 of the present invention.

[0030] Figure 3 It is a schematic diagram of an encoder module provided in Embodiment 1 of the present invention.

[0031] Figure 4 It is a schematic diagram of a cross-scale feature fusion module provided in Example 1 of the present invention.

[0032] Figure 5 It is a schematic diagram of a decoder module provided in Embodiment 1 of the present invention.

[0033] Figure 6 It is a schematic diagram of a decoding layer in the first stage provided in Embodiment 1 of the present invention.

[0034] Figure 7 It is a schematic diagram of a decoding layer in the second stage provided in the first embodiment of the present invention.

[0035] Figure 8 It is a structural schematic diagram of a target detection device provided in Embodiment 2 of the present invention.

[0036] Fig. 9 It is a structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can cover sequential implementations other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Embodiment 1

[0040] See also Figure 1 , shows a flow chart of a target detection method provided by the first embodiment of the present invention, the method can be executed by a target detection device, the target detection device can be implemented in the form of hardware and / or software, and the target detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0041] Step 101: Determine the target detection network.

[0042] In this embodiment, an object detection network may be constructed and trained based on deep learning, and the object detection network is used to perform object detection in image data using an open vocabulary as a detection target.

[0043] Among them, Figure 2As shown in the figure, the target detection network includes a text encoder Text Encoder, an image encoder ImageEncoder, a cross-modality cross-scale fusion module (Cross Modality Cross Scale Fusion Module, CMCSFM) and a detection head Head.

[0044] Step 102: Receive image data and text information.

[0045] In this embodiment, if Figure 2 As shown, image data Image as a target detection object and text information Text as an open vocabulary can be received.

[0046] The so-called open vocabulary can refer to any noun used to express the target.

[0047] In different business scenarios, the forms of image data and text information are different. For example, image data is movies and photos, and text information is search keywords, etc.

[0048] For example, in business scenarios such as video surveillance of rail transit, when an edge box receives a retrieval request for surveillance video data, multiple frames of image data can be read from the surveillance video data, and text information to be detected can be read from the retrieval request.

[0049] Among them, the edge box is an edge computing node, which is usually equipped with heterogeneous processors such as low-performance graphics processing units (GPUs) or embedded neural network processors (NPUs). For example, the performance of the T4 graphics card is 2.3FPS (Frames Per Second, the number of frames of image data processed per second) to support deep learning-based target detection networks. It can provide storage, computing, network and other resources, and sink some key business applications to the edge of the access network to reduce the width and latency losses caused by network transmission and multi-level forwarding.

[0050] Step 103: In the text encoder, encode the text information into a first semantic feature.

[0051] In this embodiment, if Figure 2 As shown, the text information Text can be input into the text encoder Text Encoder to extract features, which are recorded as the first semantic feature E t .

[0052] This process can be expressed as:

[0053] E t =Φ T (x t )

[0054] Among them, xt is the text information, Φ T is the text encoder, E t It is the first semantic feature.

[0055] Exemplarily, the text encoder may be BERT (Bidirectional Encoder Representations from Transformers), and the text information is input into BERT to extract the 256-dimensional first semantic features.

[0056] Step 104: In the image encoder, the image data is encoded into image features at multiple scales.

[0057] In this embodiment, if Figure 2 As shown, the image data Image can be input into the image encoder Image Encoder to extract multiple features of different scales, which are recorded as image features.

[0058] Exemplarily, the image encoder is a convolutional neural network (CNN), such as the EfficientVit model. Compared with the Transformer, the convolution structure in the convolutional neural network is more efficient, which can improve the reasoning efficiency of the target detection network in edge boxes with fewer resources.

[0059] At this time, in the convolutional neural network, a convolution operation is performed on the image data to encode the image data into the second semantic feature E in sequence. P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 , as image features.

[0060] Among them, the second semantic feature E P2 The scale is larger than the third semantic feature E P3 The scale of the third semantic feature E P3 The scale is larger than the fourth semantic feature E P4 The fourth semantic feature E P4 The scale is larger than the fifth semantic feature E P5 scale.

[0061] This process can be expressed as:

[0062] {E p2 ,E p3 ,E p4 ,E p5}=Φ I (x i )

[0063] Among them, x i is the image data, Φ I is the image encoder, E p2 is the second semantic feature, E p3 is the third semantic feature, E p4 is the fourth semantic feature, E p5 It is the fifth semantic feature.

[0064] For example, the resolution of the image data is 3×640×960. In the EfficientVit model, the image data is downsampled by 8 times to obtain the second semantic feature E P2 , downsample the image data by 16 times to obtain the third semantic feature E P3 , downsample the image data by 32 times to obtain the fourth semantic feature E P4 , downsample the image data by 64 times to obtain the fifth semantic feature E P5 .

[0065] Step 105: In a cross-modal and cross-scale fusion module, the first semantic feature is locally and globally interactively fused with image features of multiple scales into a multi-modal semantic feature.

[0066] like Figure 2 As shown, the first semantic feature E t and image features at multiple scales (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) is input into the cross-modal and cross-scale fusion module CMCSFM for cross-modal and cross-scale fusion, and the first semantic feature E t The image features at multiple scales are locally transformed into global interactions and fused into new features, which are recorded as multimodal semantic features.

[0067] The so-called local-to-global interactive fusion refers to the preliminary interactive fusion of the first semantic feature with some image features. When the preliminary interactive fusion is completed, the first semantic feature continues to interactively fuse with all image features to obtain a multimodal semantic feature.

[0068] In one embodiment of the present invention, Figure 2 As shown, the cross-modal cross-scale fusion module CMCSFM includes an encoder module, a cross-scale feature fusion module (Cross Scale Feature Fusion, CSFF), and a decoder module. In this embodiment, step 105 may include the following steps:

[0069] Step 1051: In the encoder module, a first fusion encoding operation is performed on the first semantic feature and the partial image feature.

[0070] In this embodiment, if Figure 2 As shown, the first semantic feature E t and some image features (such as the fifth semantic feature E P5 ) is input to perform the first fusion encoding operation in the encoder module Encoder Module, so that the first semantic feature E t and some image features (such as the fifth semantic feature E P5 ) interactive fusion, thereby preliminarily updating the first semantic feature E t and some image features (such as the fifth semantic feature E P5 ).

[0071] Exemplarily, the encoder module includes a picture-text attention module, a text attention module and a single-scale deformable attention module. Accordingly, the image features of multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature and a fifth semantic feature.

[0072] In this example, if Figure 3 As shown, the original first semantic feature E t With the original fifth semantic feature input E P5 In the image-text attention module, based on the first semantic feature E t With the fifth semantic feature E P5 The attention between them performs the image-text fusion encoding operation to transform the first semantic feature E t Fuse into the fifth semantic feature and update the fifth semantic feature E P5 .

[0073] The attention mechanism Attention is applied in the image-text attention module. At this time, the fifth semantic feature is the Query in the attention mechanism Attention, and the first semantic feature is the Key and Value. The attention weight between the two is calculated, and the first semantic feature E is added according to the attention weight. t Weighted fusion to the fifth semantic feature E P5 .

[0074] If the image-text fusion encoding operation is completed, the fifth semantic feature E output by the image-text attention module is P5 Input single-scale deformable attention module, on the fifth semantic feature E P5 Under the self-attention of P5 Execute the self-encoding operation to complete the update of the fifth semantic feature E P5 .

[0075] Among them, the self-attention mechanism SelfAttention is applied in the single-scale deformable attention module. At this time, the fifth semantic feature E P5 is the Query, Key, and Value in the self-attention mechanism, which is used to calculate the fifth semantic feature E P5 Its own self-attention weight, so as to adjust the fifth semantic feature E according to the self-attention weight P5 to encode.

[0076] The original first semantic feature E t In the input text attention module, the first semantic feature E t Under the self-attention of t Execute the self-encoding operation to complete the update of the first semantic feature E t .

[0077] Among them, the self-attention mechanism SelfAttention is applied in the text attention module. At this time, the first semantic feature E t is the Query, Key, and Value in the self-attention mechanism, which is used to calculate the first semantic feature E t Its own self-attention weight, thereby encoding the first semantic feature according to the self-attention weight.

[0078] In this example, this process can be expressed as:

[0079] {E t ,E p5}=Φ enc (E t ,E p5 0

[0080] Among them, E t is the first semantic feature, E p5 is the fifth semantic feature, Φ enc For the encoder module.

[0081] In this embodiment, attention is not calculated on the first semantic feature in the text encoder. Instead, attention is calculated on the first semantic feature when cross-modal fusion is performed in the encoder, so that the reasoning speed of the target detection network is basically not limited by the length of the first semantic feature.

[0082] Step 1052: If the first fusion coding operation is completed, a second fusion coding operation is performed on all image features in the cross-scale feature fusion module.

[0083] like Figure 2As shown, if the first fusion encoding operation is completed, the first semantic feature E t , some image features (such as the fifth semantic feature E P5 ) is updated, then all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) is input into the cross-scale feature fusion module CSFF for cross-scale feature fusion, and all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) performs the second fusion encoding operation so that all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) interactively fuses to update all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ).

[0084] Among them, all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) contains some image features that have been updated in the encoder module (such as the fifth semantic feature E P5 ) and some image features that are not updated in the encoder module (such as the second semantic feature, the third semantic feature, and the fourth semantic feature).

[0085] For example, Figure 4 As shown, the cross-scale feature fusion module CSFF includes three convolutional layers and three upsampling layers. Accordingly, the image features of multiple scales include the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 .

[0086] The convolution layer provides a downsampling operation, and the upsampling layer provides an upsampling operation. The multiples of the downsampling operation and the upsampling operation cooperate with each other. For example, the multiples of the downsampling operation and the upsampling operation are both 2 times, so as to maintain the second semantic feature EP2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 The scale remains unchanged.

[0087] Then, in this example, the fifth semantic feature E P5 In the input upsampling layer, the fifth semantic feature E P5 A first up-sampling operation Up Sample is performed.

[0088] If the first up-sampling operation Up Sample is completed, the first adding operation Add is performed to add the fifth semantic feature E after the first up-sampling operation Up Sample to the fifth semantic feature E after the first up-sampling operation Up Sample. P5 Fusion into the fourth semantic feature E P4 .

[0089] If the first addition operation Add is completed, the fourth semantic feature E after the first addition operation Add is P4 In the input upsampling layer, the fourth semantic feature E after the first addition operation Add P4 A second up-sampling operation Up Sample is performed.

[0090] If the second up-sampling operation Up Sample is completed, the second adding operation Add is performed to add the fourth semantic feature E after the second up-sampling operation Up Sample to the P4 Fusion into the third semantic feature E P3 middle.

[0091] If the second addition operation Add is completed, the third semantic feature E after the second addition operation Add is P3 In the input upsampling layer, the third semantic feature E P3 A third up-sampling operation Up Sample is performed.

[0092] If the third up-sampling operation Up Sample is completed, a third adding operation Add is performed to add the third semantic feature E after the third up-sampling operation Up Sample to the third semantic feature E after the third up-sampling operation Up Sample. P3 Fusion to the second semantic feature E P2 In the second semantic feature E P2 Updates.

[0093] If the third addition operation Add is completed, the second semantic feature E after the third addition operation Add P2 In the input convolutional layer, the second semantic feature E after the third addition operation Add P2 A first down sampling operation Down Sample is performed.

[0094] If the first down-sampling operation Down Sample is completed, the fourth addition operation Add is performed to add the second semantic feature E after the first down-sampling operation Down Sample to the P2 The third semantic feature E after fusion into the second addition operation Add P3 In the third semantic feature E P3 Updates.

[0095] If the fourth addition operation Add is completed, the third semantic feature E after the fourth addition operation Add is P3 In the input convolution layer, the third semantic feature E after the fourth addition operation P3 A second down sampling operation Down Sample is performed.

[0096] If the second down-sampling operation Down Sample is completed, the fifth addition operation Add is performed to add the third semantic feature E after the second down-sampling operation Down Sample to the P3 The fourth semantic feature E after the first addition operation Add P4 In the fourth semantic feature E P4 Updates.

[0097] If the fifth addition operation Add is completed, the fourth semantic feature E after the fifth addition operation Add is P4 In the input convolutional layer, the fourth semantic feature E after the fifth addition operation Add P4 A third down sampling operation Down Sample is performed.

[0098] If the third down-sampling operation Down Sample is completed, the sixth addition operation Add is performed to add the fourth semantic feature E after the third down-sampling operation Down Sample to the P4 Fusion to the fifth semantic feature E P5 In the fifth semantic feature E P5 Updates.

[0099] In this example, this process can be expressed as:

[0100] {E p2 ,E p3 ,E p4 ,E p5}=Φ csff (E p2 ,E p3 ,E p4 ,E p5 0

[0101] Among them, E p2 is the second semantic feature, E p3is the third semantic feature, E p4 is the fourth semantic feature, E p5 is the fifth semantic feature, Φ I It is a cross-scale feature fusion module.

[0102] Step 1053: If the second fusion encoding operation is completed, in the decoder module, the first semantic feature and the image features of multiple scales are fused and decoded from local to global into a multi-modal semantic feature.

[0103] like Figure 2 As shown, if the second combined encoding operation is completed, all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) completes the update, the first semantic feature E output by the encoder module EncoderModule can be t The image features of multiple scales (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) is input into the decoder module DecoderModule, and the first semantic feature E t and image features at multiple scales (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) From local to global fusion decoding, multimodal semantic features E 0 .

[0104] The so-called local-to-global fusion decoding refers to the first semantic feature E t Compared with the image features of some scales (such as the fifth semantic feature E P5 ) Preliminary fusion decoding. When the preliminary fusion decoding is completed, the first semantic feature E t With all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) continues to fuse the decoding to obtain multimodal semantic features.

[0105] In one embodiment of the present invention, the decoder module Decoder Module includes multiple decoding layers DecoderLayer; then, in this embodiment, step 1053 may further include the following steps:

[0106] Step 10531, divide multiple decoding layers into a first stage and a second stage in sequence.

[0107] like Figure 2 As shown, in the decoder module Decoder Module, multiple decoding layers Decoder Layer are arranged in sequence. At this time, the multiple decoding layers Decoder Layer can be divided into a first stage Stage 1 and a second stage Stage 2 while maintaining the order of sorting unchanged. The first stage Stage 1 and the second stage Stage 2 each include at least one decoding layer Decoder Layer, and the decoding layer Decoder Layer of the first stage Stage 1 is sorted before the decoding layer Decoder Layer of the second stage Stage 2.

[0108] For example, Figure 5 As shown, if the decoder module Decoder Module includes four decoding layers DecoderLayer, the first decoding layer Decoder Layer, the second decoding layer Decoder Layer, and the third decoding layer Decoder Layer can be divided into the first stage Stage1, and the fourth decoding layer Decoder Layer can be divided into the second stage Stage2.

[0109] Step 10532: Initialize multimodal semantic features.

[0110] In this embodiment, if Figure 2 and Figure 5 As shown in Figure 2, a vector of a specified size (such as 300×256) can be initialized by random or other methods as the initial multimodal semantic feature E 0 .

[0111] Step 10533: In each decoding layer of the first stage, perform a first fusion decoding operation to fuse the first semantic feature with the partial image feature into a multimodal semantic feature.

[0112] like Figure 5 As shown, in the first stage Stage1, on the one hand, the first semantic feature E output by the encoder module Encoder Module t Combined with some image features output by the cross-scale feature fusion module CSFF (such as the fifth semantic feature E P5 ) are respectively input into each decoding layer Decoder Layer of the first stage Stage1. On the other hand, the initial multimodal semantic feature E 0Input the first decoding layer Decoder Layer, and convert the multimodal semantic features E output by the previous decoding layer Decoder Layer 0 Input to the non-first decoding layer Decoder Layer.

[0113] Then, in each decoding layer of the first stage Stage1, the first fusion decoding operation can be performed to convert the first semantic feature E t and some image features (such as the fifth semantic feature E P5 ) are fused into multimodal semantic features.

[0114] For example, Figure 6 As shown in FIG. 1 , each decoding layer includes a self-attention module, a cross-modal attention module and a multi-scale deformable attention module. Accordingly, the image features of multiple scales include the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 .

[0115] Then, in each decoding layer Decoder Layer of the first stage Stage1, the first fusion decoding operation includes the following steps:

[0116] S311, input the multimodal semantic features into the self-attention module, and 0 The self-attention performs the first self-decoding operation.

[0117] Among them, the self-attention mechanism SelfAttention is applied in the self-attention module. At this time, the multimodal semantic feature E 0 is the Query, Key, and Value in the self-attention mechanism, which is used to calculate the multimodal semantic feature E 0 Its own self-attention weight, so as to adjust the multimodal semantic feature E according to the self-attention weight 0 to decode.

[0118] S312. If the first self-decoding operation is completed, the first semantic feature and the multimodal semantic feature output by the self-attention module are input into the cross-modal attention module, and a first interactive decoding operation is performed based on the attention between the multimodal semantic feature and the first semantic feature to fuse the first semantic feature into the multimodal semantic feature.

[0119] Among them, the attention mechanism Attention is applied in the cross-modal attention module. At this time, the multimodal semantic feature E 0is the Query in the attention mechanism Attention, the first semantic feature E t is Key and Value, which is used to calculate the multimodal semantic feature E 0 With the first semantic feature E t The attention weight between them is used to convert the first semantic feature E into t Weighted fusion to multimodal semantic features E 0 .

[0120] S313: If the first interactive decoding operation is completed, the fifth semantic feature E P5 The multimodal semantic feature E output by the cross-modal attention module 0 Input into the multi-scale deformable attention module, according to the multimodal semantic feature E 0 With the fifth semantic feature E P5 The attention between them performs the second interactive decoding operation to transform the fifth semantic feature E P5 Fusion to multimodal semantic features E 0 middle.

[0121] Among them, the attention mechanism Attention is applied in the multi-scale deformable attention module. At this time, the multimodal semantic feature E 0 is the Query in the attention mechanism Attention, the fifth semantic feature E P5 is Key and Value, which is used to calculate the multimodal semantic feature E 0 With the fifth semantic feature E P5 The attention weight between them is used to convert the fifth semantic feature E P5 Weighted fusion to multimodal semantic features E 0 middle.

[0122] Step 10534: If the second fusion decoding operation is completed, then in each decoding layer of the second stage, the second fusion decoding operation is performed to fuse the first semantic feature with all image features into a multimodal semantic feature.

[0123] like Figure 5 As shown, in the second stage Stage2, on the one hand, the first semantic feature E output by the encoder module Encoder Module t Combined with all image features output by the cross-scale feature fusion module CSFF (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5) are respectively input into each decoding layer Decoder Layer of the second stage Stage2. On the other hand, the multimodal semantic features E finally output by each decoding layer Decoder Layer of the first stage Stage1 are 0 Input the first decoding layer Decoder Layer, and convert the multimodal semantic features E output by the previous decoding layer Decoder Layer 0 Input to the non-first decoding layer Decoder Layer.

[0124] Then, in each decoding layer of the second stage Stage2, the second fusion decoding operation can be performed to convert the first semantic feature E t With all image features (such as the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 ) are fused into multimodal semantic features.

[0125] For example, Figure 7 As shown in FIG. 1 , each decoding layer includes a self-attention module, a cross-modal attention module and a multi-scale deformable attention module. Accordingly, the image features of multiple scales include the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 .

[0126] Then, in each decoding layer Decoder Layer of the second stage Stage2, the second fusion decoding operation includes the following steps:

[0127] S321. Input the multimodal semantic features into the self-attention module, and perform a second self-decoding operation based on the self-attention of the multimodal semantic features.

[0128] Among them, the self-attention mechanism SelfAttention is applied in the self-attention module. At this time, the multimodal semantic feature E 0 is the Query, Key, and Value in the self-attention mechanism, which is used to calculate the multimodal semantic feature E 0 Its own self-attention weight, so as to adjust the multimodal semantic feature E according to the self-attention weight 0 to decode.

[0129] S322. If the second self-decoding operation is completed, the first semantic feature and the multimodal semantic feature output by the self-attention module are input into the cross-modal attention module, and a third interactive decoding operation is performed based on the attention between the multimodal semantic feature and the first semantic feature to fuse the first semantic feature into the multimodal semantic feature.

[0130] Among them, the attention mechanism Attention is applied in the cross-modal attention module. At this time, the multimodal semantic feature E 0 is the Query in the attention mechanism Attention, the first semantic feature E t is Key and Value, which is used to calculate the multimodal semantic feature E 0 With the first semantic feature E t The attention weight between them is used to convert the first semantic feature E into t Weighted fusion to multimodal semantic features E 0 .

[0131] S323. If the third interactive decoding operation is completed, the second semantic feature, the third semantic feature, the fourth semantic feature, the fifth semantic feature and the multimodal semantic feature output by the cross-modal attention module are input into the multi-scale deformable attention module, and the second interactive decoding operation is performed in turn based on the attention between the multimodal semantic feature and the second semantic feature, the third semantic feature, the fourth semantic feature and the fifth semantic feature, so as to fuse the second semantic feature, the third semantic feature, the fourth semantic feature and the fifth semantic feature into the multimodal semantic feature in turn.

[0132] Among them, the attention mechanism Attention is applied in the multi-scale deformable attention module. At this time, the multimodal semantic feature E 0 is the Query in the attention mechanism Attention, which is followed by the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 is Key and Value, which are used to calculate the multimodal semantic features E 0 With the second semantic feature E P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5 The attention weight between them is then used to convert the second semantic feature E into P2 , the third semantic feature E P3 、The fourth semantic feature E P4 With the fifth semantic feature E P5Weighted fusion to multimodal semantic features E 0 middle.

[0133] In this example, this process can be expressed as:

[0134] E 0 =Φ dec4 (E 0 ,E p2 ,E p3 ,E p4 ,E p5 ,Φ dec1-3 (E 0 ,E p5 ,E t ))

[0135] Among them, E 0 is the multimodal semantic feature, E t is the first semantic feature, E p2 is the second semantic feature, E p3 is the third semantic feature, E p4 is the fourth semantic feature, E p5 is the fifth semantic feature, Φ dec1-3 is the first decoding layer, the second decoding layer and the third decoding layer, Φ dec4 It is the fourth decoding layer.

[0136] Step 106: In the detection head, a detection frame containing text information is generated in the image data based on the multimodal semantic features.

[0137] In practical applications, the detection head includes structures such as convolutional layer and fully connected layer (FC).

[0138] like Figure 2 As shown in the figure, the multimodal semantic features E output by the cross-modal and cross-scale fusion module CMCSFM are 0 The input detection head Head performs operations such as reducing the size and extracting features to predict the coordinates (x, y, w, h) of the detection box BBox and its confidence (Score) (a decimal between 0 and 1). At this time, the coordinates of the detection box BBox output by the detection head Head are on the image data, and its content (i.e., the pixels in the image data) is text information, thereby realizing target detection based on open vocabulary.

[0139] This process can be expressed as:

[0140] {BBox,Score}=Φ head (E 0 )

[0141] Among them, E 0is a multimodal semantic feature, BBox is a detection box, Score is the confidence, Φ head For the detection head.

[0142] In this embodiment, a target detection network is determined; the target detection network includes a text encoder, an image encoder, a cross-modal and cross-scale fusion module and a detection head; image data and text information are received; in the text encoder, the text information is encoded into a first semantic feature; in the image encoder, the image data is encoded into image features of multiple scales; in the cross-modal and cross-scale fusion module, the first semantic feature and the image features of multiple scales are locally transformed from local to global interactive fusion into a multi-modal semantic feature; in the detection head, a detection frame with text information is generated in the image data based on the multi-modal semantic feature. Using some image features in the early stage and all image features in the later stage can greatly improve the efficiency of open vocabulary-based target detection without losing image features, and is suitable for deployment in edge boxes with limited computing resources.

[0143] By applying the object detection method in this embodiment, smooth open vocabulary-based object detection can be achieved on a T4 graphics card at a performance of 30FPS.

[0144] Embodiment 2

[0145] See also Figure 8 , shows a schematic structural diagram of a target detection device provided in Embodiment 2 of the present invention.

[0146] like Figure 8 As shown, the device comprises:

[0147] A network determination module 801 is used to determine a target detection network; the target detection network includes a text encoder, an image encoder, a cross-modal and cross-scale fusion module and a detection head;

[0148] The detection information receiving module 802 is used to receive image data and text information;

[0149] A text encoding module 803, configured to encode the text information into a first semantic feature in the text encoder;

[0150] An image encoding module 804, configured to encode the image data into image features of multiple scales in the image encoder;

[0151] An image-text interactive fusion module 805 is used to interactively fuse the first semantic feature with the image features of multiple scales locally from local to global into a multimodal semantic feature in the cross-modal and cross-scale fusion module;

[0152] The detection frame output module 806 is used to generate a detection frame whose content is the text information in the image data according to the multimodal semantic features in the detection head.

[0153] In one embodiment of the present invention, the detection information receiving module 802 includes:

[0154] The monitoring information receiving module is used to read multiple frames of image data from the monitoring video data and read the text information to be detected from the retrieval request when receiving a retrieval request for monitoring video data in the edge box.

[0155] In one embodiment of the present invention, the image encoder is a convolutional neural network; the text encoding module 803 includes:

[0156] A multi-size convolution module, used for performing a convolution operation on the image data in the convolutional neural network to sequentially encode the image data into a second semantic feature, a third semantic feature, a fourth semantic feature and a fifth semantic feature as image features;

[0157] Among them, the scale of the second semantic feature is larger than the scale of the third semantic feature, the scale of the third semantic feature is larger than the scale of the fourth semantic feature, and the scale of the fourth semantic feature is larger than the scale of the fifth semantic feature.

[0158] In one embodiment of the present invention, the cross-modal and cross-scale fusion module includes an encoder module, a cross-scale feature fusion module, and a decoder module;

[0159] The image-text interaction fusion module 805 includes:

[0160] A first fusion encoding operation execution module, configured to execute a first fusion encoding operation on the first semantic feature and part of the image feature in the encoder module;

[0161] a second fusion coding operation execution module, configured to execute a second fusion coding operation on all the image features in the cross-scale feature fusion module if the first fusion coding operation is completed;

[0162] The image-text fusion decoding module is used to, if the second fusion encoding operation is completed, fuse and decode the first semantic feature with the image features of multiple scales from local to global into a multimodal semantic feature in the decoder module.

[0163] In one embodiment of the present invention, the image features at multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature;

[0164] The first fusion encoding operation execution module includes:

[0165] A graphic-text fusion coding operation execution module, configured to execute a graphic-text fusion coding operation according to the attention between the first semantic feature and the fifth semantic feature, so as to fuse the first semantic feature into the fifth semantic feature;

[0166] An image self-encoding operation execution module, configured to, if the image-text fusion encoding operation is completed, perform a self-encoding operation on the fifth semantic feature under the self-attention of the fifth semantic feature;

[0167] A text auto-encoding operation execution module is used to perform a self-encoding operation on the first semantic feature under the self-attention of the first semantic feature.

[0168] In one embodiment of the present invention, the image features at multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature;

[0169] The second fusion encoding operation execution module includes:

[0170] A first upsampling operation execution module, configured to perform a first upsampling operation on the fifth semantic feature;

[0171] a first addition operation execution module, configured to execute a first addition operation to merge the fifth semantic feature into the fourth semantic feature if the first upsampling operation is completed;

[0172] a second upsampling operation execution module, configured to execute a second upsampling operation on the fourth semantic feature if the first adding operation is completed;

[0173] a second addition operation execution module, configured to execute a second addition operation if the second upsampling operation is completed, so as to merge the fourth semantic feature into the third semantic feature;

[0174] a third upsampling operation execution module, configured to execute a third upsampling operation on the third semantic feature if the second adding operation is completed;

[0175] a third addition operation execution module, configured to execute a third addition operation if the third upsampling operation is completed, so as to merge the third semantic feature into the second semantic feature;

[0176] a first downsampling operation execution module, configured to execute a first downsampling operation on the second semantic feature if the third adding operation is completed;

[0177] a fourth addition operation execution module, configured to execute a fourth addition operation if the first downsampling operation is completed, so as to merge the second semantic feature into the third semantic feature;

[0178] a second downsampling operation execution module, configured to execute a second downsampling operation on the third semantic feature if the fourth adding operation is completed;

[0179] a fifth adding operation executing module, configured to execute a fifth adding operation if the second downsampling operation is completed, so as to merge the third semantic feature into the fourth semantic feature;

[0180] a third downsampling operation execution module, configured to execute a third downsampling operation on the fourth semantic feature if the fifth adding operation is completed;

[0181] The sixth adding operation execution module is configured to execute a sixth adding operation if the third down-sampling operation is completed, so as to merge the fourth semantic feature into the fifth semantic feature.

[0182] In one embodiment of the present invention, the decoder module includes multiple decoding layers; the image-text fusion decoding module includes:

[0183] A stage division module, used for dividing the plurality of decoding layers into a first stage and a second stage in sequence;

[0184] A multimodal semantic feature initialization module, used to initialize multimodal semantic features;

[0185] A first-stage decoding module, configured to perform a first fusion decoding operation in each of the decoding layers of the first stage, so as to fuse the first semantic feature and part of the image feature into the multimodal semantic feature;

[0186] The second stage decoding module is used to perform a second fusion decoding operation in each of the decoding layers of the second stage if the second fusion decoding operation is completed, so as to fuse the first semantic feature and all the image features into the multimodal semantic feature.

[0187] In one embodiment of the present invention, the image features at multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature;

[0188] The first stage decoding module is also used for:

[0189] In each of the decoding layers of the first stage, a first self-decoding operation is performed according to the self-attention of the multimodal semantic feature, and if the first self-decoding operation is completed, a first interactive decoding operation is performed according to the attention between the multimodal semantic feature and the first semantic feature to fuse the first semantic feature into the multimodal semantic feature, and if the first interactive decoding operation is completed, a second interactive decoding operation is performed according to the attention between the multimodal semantic feature and the fifth semantic feature to fuse the fifth semantic feature into the multimodal semantic feature;

[0190] The second stage decoding module is also used to include:

[0191] In each decoding layer of the second stage, a second self-decoding operation is performed based on the self-attention of the multimodal semantic feature. If the second self-decoding operation is completed, a third interactive decoding operation is performed based on the attention between the multimodal semantic feature and the first semantic feature to fuse the first semantic feature into the multimodal semantic feature. If the third interactive decoding operation is completed, a second interactive decoding operation is performed based on the attention between the multimodal semantic feature and the second semantic feature, the third semantic feature, the fourth semantic feature, and the fifth semantic feature in sequence to fuse the second semantic feature, the third semantic feature, the fourth semantic feature, and the fifth semantic feature in sequence into the multimodal semantic feature.

[0192] The target detection device provided in the embodiment of the present invention can execute the target detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the target detection method.

[0193] Embodiment 3

[0194] See also Fig. 9 , shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0195] like Fig. 9As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0196] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0197] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the target detection method.

[0198] In some embodiments, the target detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the target detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the target detection method in any other appropriate manner (e.g., by means of firmware).

[0199] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0200] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0201] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0202] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0203] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0204] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0205] Embodiment 4

[0206] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the target detection method provided by any embodiment of the present invention is implemented.

[0207] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0208] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0209] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A target detection method, characterized in that: include: Determine the target detection network; The object detection network includes a text encoder, an image encoder, a cross-modal and cross-scale fusion module and a detection head; receiving image data and text information; In the text encoder, encoding the text information into a first semantic feature; In the image encoder, encoding the image data into image features at multiple scales; In the cross-modal and cross-scale fusion module, the first semantic feature and the image features of multiple scales are locally and globally interactively fused into a multi-modal semantic feature; In the detection head, a detection frame containing the text information is generated in the image data according to the multimodal semantic features.

2. The method according to claim 1, characterized in that The receiving of image data and text information includes: When receiving a search request for surveillance video data, the edge box reads multiple frames of image data from the surveillance video data, and reads text information to be detected from the search request.

3. The method according to claim 1, characterized in that The image encoder is a convolutional neural network; in the image encoder, the image data is encoded into image features of multiple scales, including: In the convolutional neural network, a convolution operation is performed on the image data to sequentially encode the image data into a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature as image features; Among them, the scale of the second semantic feature is larger than the scale of the third semantic feature, the scale of the third semantic feature is larger than the scale of the fourth semantic feature, and the scale of the fourth semantic feature is larger than the scale of the fifth semantic feature.

4. The method according to any one of claims 1 to 3, characterized in that The cross-modal and cross-scale fusion module includes an encoder module, a cross-scale feature fusion module, and a decoder module; In the cross-modal and cross-scale fusion module, the first semantic feature and the image features of multiple scales are locally and globally interactively fused into a multi-modal semantic feature, including: In the encoder module, performing a first fusion encoding operation on the first semantic feature and part of the image feature; If the first fusion encoding operation is completed, then in the cross-scale feature fusion module, a second fusion encoding operation is performed on all the image features; If the second fusion encoding operation is completed, the first semantic feature and the image features at multiple scales are fused and decoded from local to global into a multimodal semantic feature in the decoder module.

5. The method according to claim 4, characterized in that The image features at multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature; In the encoder module, performing a first fusion encoding operation on the first semantic feature and part of the image feature includes: Performing a graphic-text fusion encoding operation according to the attention between the first semantic feature and the fifth semantic feature to fuse the first semantic feature into the fifth semantic feature; If the image-text fusion encoding operation is completed, then under the self-attention of the fifth semantic feature, a self-encoding operation is performed on the fifth semantic feature; Under the self-attention of the first semantic feature, a self-encoding operation is performed on the first semantic feature.

6. The method according to claim 4, characterized in that The image features at multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature; In the cross-scale feature fusion module, performing a second fusion encoding operation on all the image features includes: Performing a first upsampling operation on the fifth semantic feature; If the first upsampling operation is completed, performing a first adding operation to merge the fifth semantic feature into the fourth semantic feature; If the first adding operation is completed, performing a second upsampling operation on the fourth semantic feature; If the second upsampling operation is completed, performing a second addition operation to merge the fourth semantic feature into the third semantic feature; If the second adding operation is completed, performing a third upsampling operation on the third semantic feature; If the third upsampling operation is completed, performing a third adding operation to merge the third semantic feature into the second semantic feature; If the third adding operation is completed, performing a first downsampling operation on the second semantic feature; If the first downsampling operation is completed, performing a fourth adding operation to merge the second semantic feature into the third semantic feature; If the fourth adding operation is completed, performing a second downsampling operation on the third semantic feature; If the second downsampling operation is completed, performing a fifth adding operation to merge the third semantic feature into the fourth semantic feature; If the fifth adding operation is completed, performing a third downsampling operation on the fourth semantic feature; If the third down-sampling operation is completed, a sixth adding operation is performed to merge the fourth semantic feature into the fifth semantic feature.

7. The method according to claim 4, characterized in that The decoder module includes a plurality of decoding layers; in the decoder module, the first semantic feature and the image features of multiple scales are fused and decoded from local to global into a multimodal semantic feature, including: Dividing the plurality of decoding layers into a first stage and a second stage in sequence; Initialize multimodal semantic features; In each of the decoding layers of the first stage, performing a first fusion decoding operation to fuse the first semantic feature and part of the image feature into the multimodal semantic feature; If the second fusion decoding operation is completed, a second fusion decoding operation is performed in each of the decoding layers of the second stage to fuse the first semantic feature and all the image features into the multimodal semantic feature.

8. The method according to claim 7, characterized in that The image features at multiple scales include a second semantic feature, a third semantic feature, a fourth semantic feature, and a fifth semantic feature; In each of the decoding layers of the first stage, performing a first fusion decoding operation to fuse the first semantic feature and part of the image feature into the multimodal semantic feature includes: In each of the decoding layers of the first stage, a first self-decoding operation is performed according to the self-attention of the multimodal semantic feature, and if the first self-decoding operation is completed, a first interactive decoding operation is performed according to the attention between the multimodal semantic feature and the first semantic feature to fuse the first semantic feature into the multimodal semantic feature, and if the first interactive decoding operation is completed, a second interactive decoding operation is performed according to the attention between the multimodal semantic feature and the fifth semantic feature to fuse the fifth semantic feature into the multimodal semantic feature; In each of the decoding layers of the second stage, performing a second fusion decoding operation to fuse the first semantic feature and all the image features into the multimodal semantic feature includes: In each decoding layer of the second stage, a second self-decoding operation is performed based on the self-attention of the multimodal semantic feature. If the second self-decoding operation is completed, a third interactive decoding operation is performed based on the attention between the multimodal semantic feature and the first semantic feature to fuse the first semantic feature into the multimodal semantic feature. If the third interactive decoding operation is completed, a second interactive decoding operation is performed based on the attention between the multimodal semantic feature and the second semantic feature, the third semantic feature, the fourth semantic feature, and the fifth semantic feature in sequence to fuse the second semantic feature, the third semantic feature, the fourth semantic feature, and the fifth semantic feature in sequence into the multimodal semantic feature.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the target detection method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target detection method according to any one of claims 1 to 8 is implemented.

Images