Object Detection Method, Device, Equipment and Storage Medium

Abstract

The present application discloses a target detection method, apparatus, device and storage medium. The method includes obtaining multi-scale image features of an image to be detected and text features of a preset prompt text; interacting the high-level semantic image features with the text features to obtain updated high-level semantic image features and updated text features; determining multi-modal fusion features based on the updated high-level semantic image features, the updated text features and low-level semantic image features; and determining detection information of the image to be detected based on the multi-modal fusion features. After obtaining the multi-scale image features, the present application divides the multi-scale image features into high-level semantic image features and low-level semantic image features, and then only interacts the high-level semantic image features with the text features, avoiding the huge computational amount brought by the low-level semantic image features and improving the inference speed of target detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly relates to an object detection method, device, equipment, and storage medium. Background Art

[0002] The key to an open-set object detection method or an open-set image segmentation method lies in introducing a language model to process unseen objects. Among them, the open-set object detection method reformulates object detection as a phrase alignment task and introduces contrastive training between object regions and language phrases; the open-set image segmentation method introduces CLIP to enable interaction between image features and language features, enabling a multi-modal decoder for context to achieve multi-modal fusion.

[0003] However, existing open-set object detection methods or open-set image segmentation methods all perform multi-modal fusion of text features and image features. Multi-modal fusion often requires a large amount of computation, increasing the inference time of open-set object detection or open-set image segmentation, and also having high requirements for hardware resources.

[0004] Therefore, the existing technology still needs to be improved. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide an object detection method, device, equipment, and storage medium in view of the deficiencies of the existing technology.

[0006] To solve the above technical problem, a first aspect of this application provides an object detection method, where the object detection method includes:

[0007] Obtain multi-scale image features of the image to be detected and text features of a preset prompt text;

[0008] Interact the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features;

[0009] Based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features other than the high-level semantic image features, determine multi-modal fusion features;

[0010] Based on the multi-modal fusion features, determine the detection information of the image to be detected.

[0011] For the object detection method, before the step of interacting the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features, the method further includes:

[0012] The image resolution for obtaining image features at each scale;

[0013] Based on the image resolution, divide the multi-scale image features into high-level semantic image features and low-level semantic image features, where the resolution of any high-level semantic image feature is less than that of any low-level semantic image feature.

[0014] The target detection method, wherein the interaction between the high-level semantic image features in the multi-scale image features and the text features to obtain updated high-level semantic image features and updated text features specifically includes:

[0015] Construct a query vector based on the high-level semantic image features, interact and learn the high-level semantic image features with the text features to obtain updated high-level semantic image features;

[0016] Using the text features as query vectors, interact and learn the text features with the updated high-level semantic image features to obtain updated text features.

[0017] The target detection method, wherein the construction of a query vector based on the high-level semantic image features, and the interaction and learning of the high-level semantic image features with the text features to obtain updated high-level semantic image features specifically includes:

[0018] Perform self-attention learning on the high-level semantic image features to obtain intermediate high-level semantic image features;

[0019] Construct a query vector with the intermediate high-level semantic image features, construct value vectors and key vectors with the text features, and perform cross-attention learning based on the query vector, value vectors and key vectors to obtain updated high-level semantic image features.

[0020] The target detection method, wherein the interaction between the high-level semantic image features in the multi-scale image features and the text features to obtain updated high-level semantic image features and updated text features specifically includes:

[0021] Using the text features as query vectors, interact and learn the text features with the high-level semantic image features to obtain updated text features;

[0022] Construct a query vector based on the high-level semantic image features, interact and learn the high-level semantic image features with the updated text features to obtain updated high-level semantic image features.

[0023] The target detection method described above, wherein determining the multimodal fusion feature based on the updated high-level semantic image feature, the updated text feature, and the low-level semantic image feature other than the high-level semantic image feature in the multi-scale image feature specifically includes:

[0024] Taking the image feature with the smallest resolution among the updated high-level semantic image feature and the low-level semantic image feature as the first target feature, and the text feature as the second target feature for feature fusion to obtain the first target fusion feature;

[0025] Taking the image feature with the second smallest resolution among the updated high-level semantic image feature and the low-level semantic image feature as the first target feature, and the first target fusion feature as the second target feature for feature fusion to obtain the second target fusion feature;

[0026] And so on until the Nth target fusion feature is obtained, where N is the number of features of the multi-scale image feature;

[0027] Concatenating the first target fusion feature to the Nth target fusion feature to obtain the multimodal fusion feature.

[0028] The target detection method described above, wherein the fusion process of the feature fusion specifically includes:

[0029] Determining a concatenated feature based on the first target feature and the second target feature;

[0030] Constructing a first intermediate feature and a second intermediate feature based on the concatenated feature, and determining a third intermediate feature based on the first intermediate feature;

[0031] Adding and flattening the second intermediate feature and the third intermediate feature to fuse the first target feature and the second target feature.

[0032] The target detection method described above, wherein determining the third intermediate feature based on the first intermediate feature specifically is:

[0033] Inputting the first intermediate feature into a reparameterization module, and outputting the third intermediate feature through the reparameterization module, wherein the network parameters of the reparameterization module fuse the convolutional layer parameters and the normalization layer parameters, and the convolutional calculation and the normalization operation are realized through one-step calculation.

[0034] The target detection method described above, wherein determining the concatenated feature based on the first target feature and the second target feature specifically includes:

[0035] For the case where the image feature with the smallest resolution is used as the first target feature, concatenating the first target feature and the second target feature to obtain the concatenated feature;

[0036] For cases other than when the image feature with the smallest resolution is divided as the first target feature, the second target feature is upsampled to obtain an upsampled image feature, and the upsampled image feature is concatenated with the first target feature to obtain a concatenated feature, where the image size of the upsampled image feature is the same as the feature size of the first target feature.

[0037] The second aspect of the present application provides an object detection device, where the object detection device specifically includes:

[0038] A feature extraction module, configured to obtain multi-scale image features of an image to be detected and text features of a preset prompt text;

[0039] A multi-modal feature fusion module, configured to interact the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features, and determine multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features other than the high-level semantic image features in the multi-scale image features;

[0040] A decoding module, configured to determine detection information of the image to be detected based on the multi-modal fusion features.

[0041] The third aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any one of the above object detection methods.

[0042] The fourth aspect of the present application provides a terminal device, which includes: a processor and a memory;

[0043] The memory stores a computer-readable program executable by the processor;

[0044] When the processor executes the computer-readable program, it implements the steps in any one of the above object detection methods.

[0045] Beneficial effects:

[0046] 1. After obtaining the multi-scale image features, the present application divides the multi-scale image features into high-level semantic image features and low-level semantic image features, and then only interacts the high-level semantic image features with the text features, avoiding the huge computational amount brought by the low-level semantic image features and improving the inference speed of object detection.

[0047] 2. After obtaining the updated high-level semantic image features and updated text features, the present application fuses the updated high-level semantic image features, updated text features, and low-level semantic image features, which can retain more detailed information, thereby ensuring the accuracy of the detected information.

[0048] 3. The present application fuses the updated high-level semantic image features, updated text features, and low-level semantic image features. By adopting a reparameterization module that fuses the convolutional layer parameters and normalization layer parameters, the fusion process is lightweight, which can further reduce the computational amount required for multimodal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following-described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0050] Figure 1 It is a principle block diagram of an example diagram of the object detection model.

[0051] Figure 2 It is a principle block diagram of the multimodal fusion module.

[0052] Figure 3 It is a principle block diagram of the lightweight multi-scale fusion unit.

[0053] Figure 4 It is a principle block diagram of the fusion module.

[0054] Figure 5 It is a principle block diagram of the bidirectional cross-attention layer.

[0055] Figure 6 It is a flowchart of the object detection method provided by the embodiment of the present application.

[0056] Figure 7 It is a principle flowchart of the object detection method provided by the embodiment of the present application.

[0057] Figure 8 It is a principle block diagram of the object detection device provided by the embodiment of the present application.

[0058] Figure 9 It is a principle block diagram of the terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] Embodiments of the present application provide an object detection method, apparatus, device, and storage medium. To make the objectives, technical solutions, and effects of the present application clearer and more explicit, the following further describes the present application in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more of the associated listed items.

[0061] Those skilled in the art of the present technology can understand that unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0062] It should be understood that the sequence numbers and magnitudes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0063] The following further describes the content of the application by describing the embodiments in conjunction with the accompanying drawings.

[0064] This embodiment provides an object detection method. The object detection method applies an object detection model, and the object detection model is used for open-set object detection, such as Figure 1As shown, the boot target detection model may include a feature extraction module, a multi-modal feature fusion module, and a decoding module (such as, a Transformer decoder, etc.). The feature extraction module includes a parallel image feature extraction module and a text feature extraction module. The image feature extraction module and the text feature extraction module are both connected to the multi-modal feature fusion module, and the multi-modal feature fusion module is connected to the decoding module. The image feature extraction module is used to obtain multi-scale image features of the image to be detected, the text feature extraction module is used to obtain text features of the preset prompt text, the multi-modal feature fusion module is used to fuse the multi-scale image features and the text features to determine multi-modal fusion features, and the decoding module is used to determine detection information based on the multi-modal fusion features.

[0065] In one implementation, as Figure 2 shown, the multi-modal feature fusion module includes a bidirectional cross-attention layer and a lightweight multi-scale fusion unit. The bidirectional cross-attention layer is used to interact the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features; the lightweight multi-scale fusion unit is used to determine multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features except the high-level semantic image features.

[0066] Further, as Figure 3 shown, the lightweight multi-scale fusion unit includes a plurality of cascaded fusion modules in sequence and a splicing layer. The plurality of fusion modules are all connected to the splicing layer. Among them, as Figure 4 shown, the fusion module may include two parallel convolutional layers, a reparameterization module, an adder, and a flattening layer. One of the two convolutional layers is connected to the reparameterization module, the other convolutional layer is connected to the adder, the reparameterization module is connected to the adder, and the adder is connected to the flattening layer. The fusion module fuses the input feature 1 and feature 2 to obtain a fusion feature.

[0067] As Figure 5As shown in the figure, the bidirectional cross-attention layer may include a self-attention layer and two cascaded cross-attention layers. The self-attention layer is connected to the preceding cross-attention layer. Among them, the cross-attention layer is used for self-attention learning of image features, the preceding cross-attention layer is used for cross-attention learning of image features based on text features, and the subsequent cross-attention layer is used for cross-attention learning of text features based on image features. For this purpose, the image features are used as the query vector, value vector, and key vector of the self-attention layer; the output item of the cross-attention layer is used as the query vector of the preceding cross-attention layer, the text features are used as the value vector and key vector of the preceding cross-attention layer; the output item of the preceding cross-attention layer is used as the updated image features, as well as the value vector and key vector of the subsequent cross-attention layer, and the text features are used as the query vector of the subsequent cross-attention layer.

[0068] It should be noted that in practical applications, the object detection model can also adopt other structures. For example, the bidirectional cross-attention layer includes two cascaded cross-attention layers, or the fusion module includes an adder, etc.

[0069] The above completes the description of the model structure of the object detection model. Next, the specific process of determining the detection information based on the object detection model will be described.

[0070] As Figure 6 and Figure 7 shown, the object detection method provided by the embodiments of the present application specifically includes:

[0071] S10. Obtain multi-scale image features of the image to be detected and text features of a preset prompt text.

[0072] Specifically, the preset prompt text is a description text of the detection target. For example, the preset prompt text can be a name description of the detection target, a form description of the detection target, and / or a category description of the detection target, etc. That is to say, the description information of the detection target is informed to the object detection model through the preset prompt text, so that the object detection model can know the detection target. The text features are obtained by a text feature extraction module. Among them, the text feature extraction module can segment the preset prompt text to obtain several words, then determine the word embedding of each word, and finally splice the word embeddings of each word to form a multi-dimensional text feature vector, and use this multi-dimensional text feature vector as the text features, etc.

[0073] The multi-scale image features are extracted by the image feature extraction module in the object detection model. Among them, the image feature extraction module can adopt VIT, ResNet, InceptionNet, etc. The multi-scale image features include image features with different scales of multiple images. For example, the multi-scale image features include image features of 4 scales, and the 4 scales are H1×W1×C, H2×W2×C, H3×W3×C, H4×W4×C respectively. H1, H2, H3, and H4 are all the heights of the picture features, W1, W2, W3, and W4 are all the widths of the picture features, and C is the feature dimension.

[0074] S20. Interact the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features.

[0075] Specifically, the multi-scale image features include image features of multiple different scales. Among the image features of multiple different scales, the image features of the high-resolution scale contain more detailed information and less semantic information, and the image features of the low-resolution scale contain more semantic information and less detailed information. At the same time, the number of tokens included in the image features of the high-resolution scale is more than that of the low-resolution scale image features. Therefore, when the image features of the high-resolution scale interact with the text features, a huge amount of computation will be generated, which will increase the interaction time. For this reason, in the embodiments of the present application, when the image features and the text features are interacted, only the high-level semantic image features in the multi-scale image features are interacted with the text features, and the low-level semantic image features are not interacted with the text features, so as to save the computation generated when the semantic image features (i.e., the high-resolution scale image features) interact with the text, and reduce the computation of object detection.

[0076] [[ID=*9]]In order to only interact the high-level semantic image features with the text features, after the multi-scale image features are obtained, the multi-scale image features will be divided into high-level semantic image features and low-level semantic image features, and then the high-level semantic image features in the multi-scale image features will be interacted with the text features. Based on this, before the high-level semantic image features in the multi-scale image features are interacted with the text features to obtain updated high-level semantic image features and updated text features, the method further includes:

[0077] Obtain the image resolution of each scale of image features;

[0078] Divide the multi-scale image features into high-level semantic image features and low-level semantic image features based on the image resolution.

[0079] Specifically, the high-level semantic image features may include one or more image features, and the low-level semantic image features may also include one or more image features. Moreover, the resolution of any high-level semantic image feature is less than that of any low-level semantic image feature. That is to say, the maximum resolution of the high-level semantic image features is less than the minimum resolution of the low-level semantic image features. For example, the multi-scale image features include image feature A, image feature B, image feature C, and image feature D. The high-level semantic image features include image feature A and image feature B, and the low-level semantic image features include image feature C and image feature D. Then the resolution of image feature A is less than that of image feature C and image feature D, and the resolution of image feature B is less than that of image feature C and image feature D.

[0080] Furthermore, when dividing the multi-scale image features based on the image resolution, the first quantity of the high-level semantic image features can be obtained first, and then the first quantity of image features can be selected from the multi-scale image features in ascending order of resolution as the high-level semantic image features, and the other unselected image features are used as the low-level semantic image features. Herein, the first quantity of the high-level semantic image features can be preset (e.g., 1, 2, etc., and the first quantity is less than the second quantity), or it can be determined based on the second quantity of the multi-scale image features and the preset quantities of the high-level semantic image features and the low-level semantic image features. For example, the first quantity is preset to 1, that is to say, the image feature with the smallest resolution is selected from the multi-scale image features as the high-level semantic image feature, and the remaining image features are used as the low-level semantic image features.

[0081] After obtaining the high-level semantic image features and the low-level semantic image features by division, the high-level semantic image features are interacted with the text features. And when interacting the high-level semantic image features and the text features, bidirectional cross-attention learning can be performed to provide the semantic information in the high-level semantic image features and the text features. Based on this, the interaction of the high-level semantic image features in the multi-scale image features with the text features to obtain the updated high-level semantic image features and the updated text features specifically includes:

[0082] Construct a query vector based on the high-level semantic image features, interact and learn the high-level semantic image features with the text features to obtain the updated high-level semantic image features;

[0083] Use the text features as the query vector, interact and learn the text features with the updated high-level semantic image features to obtain the updated text features.

[0084] Specifically, such as Figure 3As shown, the interactive learning between the high-level semantic image features and the text features, and the interactive learning between the text features and the updated high-level semantic image features can both be achieved through a cross-attention layer. That is, a query vector is constructed from the high-level semantic image features, value vectors and key vectors are constructed from the text features, and then the query vector, value vectors and key vectors are input into the cross-attention layer. The updated query vector is output through the cross-attention layer, and the updated query vector is used as the updated high-level semantic image features, thus realizing the interactive learning between the high-level semantic image features and the text features. Then, a query vector is constructed from the text features, value vectors and key vectors are constructed from the updated high-level semantic image features, the query vector, value vectors and key vectors are input into the cross-attention layer, the updated query vector is output through the cross-attention layer, and the updated query vector is used as the updated text features, realizing the update of the high-level semantic image features and the text features.

[0085] Of course, in practical applications, the interactive learning between the text features and the updated high-level semantic image features can also be carried out first, and then the interactive learning between the high-level semantic image features and the text features.

[0086] Based on this, in the target detection method, the step of interacting the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features specifically includes: using the text features as the query vector, performing interactive learning between the text features and the high-level semantic image features to obtain updated text features; constructing a query vector based on the high-level semantic image features, and performing interactive learning between the high-level semantic image features and the updated text features to obtain updated high-level semantic image features.

[0087] Furthermore, before performing the interactive learning, self-attention learning can be performed on the high-level semantic image features to strengthen the semantic information of the high-level semantic image features. Based on this, the step of constructing a query vector based on the high-level semantic image features, performing interactive learning between the high-level semantic image features and the text features to obtain updated high-level semantic image features specifically includes:

[0088] Performing self-attention learning on the high-level semantic image features to obtain intermediate high-level semantic image features;

[0089] Constructing a query vector from the intermediate high-level semantic image features, constructing value vectors and key vectors from the text features, and performing cross-attention learning based on the query vector, value vectors and key vectors to obtain updated high-level semantic image features.

[0090] Specifically, as Figure 3As shown, self-attention learning can be achieved through a self-attention layer, which is connected to a cross-attention layer for interactive learning between high-level semantic image features and the text features. The intermediate high-level semantic image features output by the self-attention layer are used as the query vectors of the cross-attention layer. That is to say, the high-level semantic image features are used as the query vectors, value vectors, and key vectors, and then the query vectors, value vectors, and key vectors are input into the self-attention layer. The self-attention layer performs self-attention learning on the high-level semantic image features to obtain intermediate high-level semantic image features.

[0091] It should be noted that when there is one high-level semantic image feature, the self-attention layer is configured with a self-attention mechanism. When there are multiple high-level semantic image features, the self-attention layer is configured with a variable self-attention mechanism to achieve self-attention learning of high-level semantic image features at different scales. Of course, when there are multiple high-level semantic image features, each high-level semantic image feature will interact with the text features to obtain its corresponding updated high-level semantic image feature. The text features can interact only with the high-level semantic image feature with the lowest resolution to obtain the updated text features, or can interact with each high-level semantic image feature, and then fuse all the updated text features obtained by the interaction as the updated text features, etc.

[0092] In a typical implementation manner of the embodiment of the present application, there is one high-level semantic image feature, and the self-attention layer is configured with a self-attention mechanism. This can avoid the use of variable operators and reduce the computational complexity of self-attention learning. At the same time, it can also avoid the extremely large computational complexity and the introduced background noise caused by high-resolution low-semantic image features, further improving the detection speed of the object detection and the accuracy of the detected detection information.

[0093] S30. Determine the multi-modal fusion feature based on the updated high-level semantic image feature, the updated text feature, and the low-level semantic image feature other than the high-level semantic image feature in the multi-scale image feature.

[0094] Specifically, the multi-modal fusion feature fuses the feature information of the updated high-level semantic image feature, the updated text feature, and the low-level semantic image feature, provides detailed information through the low-level semantic image feature, and performs secondary fusion on the text features, which can ensure the semantic information and detailed information included in the multi-modal fusion feature, guarantee the model performance of the object detection model, and further improve the accuracy of the detected detection information.

[0095] Further, when fusing the updated high-level semantic image features, the updated text features, and the low-level semantic image features, the fusion can be performed step by step in the order of increasing resolution of the image features, and the fused features obtained from the step-by-step fusion are concatenated to obtain the multi-modal fusion features. Based on this, determining the multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features other than the high-level semantic image features in the multi-scale image features specifically includes:

[0096] Taking the image feature with the smallest resolution among the updated high-level semantic image features and the low-level semantic image features as the first target feature, and the text feature as the second target feature for feature fusion to obtain the first target fusion feature;

[0097] Taking the image feature with the second smallest resolution among the updated high-level semantic image features and the low-level semantic image features as the first target feature, and the first target fusion feature as the second target feature for feature fusion to obtain the second target fusion feature;

[0098] And so on until the Nth target fusion feature is obtained, where N is the number of features of the multi-scale image features;

[0099] Concatenating the first target fusion feature to the Nth target fusion feature to obtain the multi-modal fusion features.

[0100] Specifically, the image feature with the smallest resolution is fused with the text feature to obtain the first target fusion feature, and then the first target fusion feature is fused with the image feature with the second smallest resolution to obtain the second target fusion feature, and so on to obtain the Nth target fusion feature.

[0101] As Figure 3 shown, the image feature with the smallest resolution and the text feature are the input items of the foremost fusion module, the output item of the foremost fusion module and the image feature with the second smallest resolution are the input items of the second fusion module, the output item of the second fusion module and the image feature with the third smallest resolution are the input items of the third fusion module, and so on, the output item of the (N - 1)th fusion module and the image feature with the Nth smallest resolution are the input items of the Nth fusion module. Through N fusion modules, N target fusion features can be obtained, and then the N fusion features are input into the concatenation layer, and the N target fusion features are concatenated through the concatenation layer to obtain the multi-modal fusion features.

[0102] Further, the fusion process of the feature fusion specifically includes:

[0103] Determining the concatenated feature based on the first target feature and the second target feature;

[0104] Construct a first intermediate feature and a second intermediate feature based on the splicing feature, and determine a third intermediate feature based on the first intermediate feature;

[0105] Add and flatten the second intermediate feature and the third intermediate feature to fuse the first target feature and the second target feature.

[0106] Since the scales of the respective image features are different, the image scales of the first target feature and the second target feature are different. Therefore, before splicing the first target feature and the second target feature, it is necessary to upsample the first target feature so that the image scale of the upsampled first target feature is the same as the image scale of the second target feature. Specifically, for the case where the image feature with the smallest resolution is the first target feature, the first target feature and the second target feature are spliced to obtain a splicing feature; for cases other than the image feature with the smallest resolution being the first target feature, the second target feature is upsampled to obtain an upsampled image feature, and the upsampled image feature and the first target feature are spliced to obtain a splicing feature, where the image size of the upsampled image feature is the same as the feature size of the first target feature.

[0107] The first intermediate feature and the second intermediate feature are obtained by processing the splicing feature. For example, as Figure 4 shown, the splicing feature is respectively input into two 1×1 convolutional layers, and the first intermediate feature and the second intermediate feature are determined through the two 1×1 convolutional layers. Among them, the two 1×1 convolutional layers can be configured with different convolutional parameters so that the first intermediate feature and the second intermediate feature determined through the two 1×1 convolutional layers are different. Of course, in practical applications, the two 1×1 convolutional layers can be configured with the same convolutional parameters, or alternatively, other methods can be used to determine the first intermediate feature and the second intermediate feature, such as obtaining the first intermediate feature by passing the splicing feature through a 1×1 convolutional layer and using the splicing feature as the second intermediate feature, etc.

[0108] The third intermediate feature is obtained by processing the first intermediate feature, for example, performing convolution operation and normalization operation on the first intermediate feature to obtain the third intermediate feature, or performing normalization operation on the first intermediate feature to obtain the third intermediate feature, etc. In an embodiment of the present application, the third intermediate feature is obtained by performing re-parameter calculation on the first intermediate feature, wherein re-parameter calculation refers to using network parameters that fuse convolution layer parameters and normalization layer parameters to calculate the first intermediate feature, that is, the convolution calculation and normalization operation of the first intermediate feature are realized through one-step calculation, which avoids repeated calculations in convolution calculation and normalization operation, and further reduces the amount of calculation in the fusion process. Specifically, a re-parameter module can be constructed, which is obtained by fusing the convolution layer and the BN layer into a new convolution layer, and the network parameters of the new convolution layer are obtained by fusing the convolution layer parameters and the BN layer parameters, that is, the calculation process of the re-parameter module can be expressed as:

[0109] f′=W·f+W

[0110] W=W bn W conv

[0111] b=W bn b conv +b bn

[0112] Among them, W and b represent the network parameters of the parameter module, W conv and b conv Represents the network parameters of the fused convolutional layer, W bn and b bn Represents the network parameters of the fused BN layer.

[0113] S40: Determine detection information of the image to be detected based on the multimodal fusion features.

[0114] After obtaining the multimodal fusion features, target detection is performed based on the multimodal fusion features to obtain detection information of the image to be detected. For example, a decoder can be used to parse the multimodal fusion features to obtain detection information of the image to be detected. Specifically, a learnable query vector is first obtained, and the learnable query vector is updated through multiple decoding layers in the decoder using the multimodal fusion features. The updated learnable query vector is then used as the learnable query vector of the next decoding layer, and the learnable query vector is updated through the decoding layer in the decoder using the multimodal fusion features. This is repeated until the last decoding layer to obtain the target query vector. Finally, the target query vector is used to perform target detection using MLP to obtain detection information of the image to be detected.

[0115] Further, the detection information may include the detection box of the detection target, or the target segmentation result of the detection target. When the detection information is the target segmentation result of the detection target, the detection box of the detection target can be obtained first, and then the detection box is multiplied by the image features of the highest resolution to obtain the target segmentation result.

[0116] In summary, this embodiment provides a target detection method, which includes obtaining multi-scale image features of an image to be detected and text features of a preset prompt text; interacting the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features; determining multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features except the high-level semantic image features; and determining the detection information of the image to be detected based on the multi-modal fusion features. After obtaining the multi-scale image features in this application, the multi-scale image features are divided into high-level semantic image features and low-level semantic image features, and then only the high-level semantic image features are interacted with the text features, avoiding the huge computational amount brought by the low-level semantic image features and improving the inference speed of target detection. At the same time, in this application, after obtaining the updated high-level semantic image features and updated text features, the updated high-level semantic image features, the updated text features, and the low-level semantic image features are fused, which can retain more detailed information, thereby ensuring the accuracy of the detected detection information.

[0117] Based on the above target detection method, this embodiment provides a target detection device, as Figure 8 shown. The target detection device specifically includes:

[0118] A feature extraction module 100, configured to obtain multi-scale image features of an image to be detected and text features of a preset prompt text;

[0119] A multi-modal feature fusion module 200, configured to interact the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features, and determine multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features except the high-level semantic image features;

[0120] A decoding module 300, configured to determine the detection information of the image to be detected based on the multi-modal fusion features.

[0121] Based on the above object detection method, this embodiment provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the object detection method as described in the above embodiment.

[0122] Based on the above object detection method, the present application further provides a terminal device, as Figure 9 shown, which includes at least one processor 20; a display screen 21; and a memory 22, and may further include a communication interface 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22, and the communication interface 23 can complete mutual communication through the bus 24. The display screen 21 is set to display a user guidance interface preset in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logical instructions in the memory 22 to execute the method in the above embodiment.

[0123] In addition, when the logical instructions in the above-mentioned memory 22 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0124] The memory 22, as a computer-readable storage medium, can be set to store software programs and computer-executable programs, such as the program instructions or modules corresponding to the method in the embodiment of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, that is, implements the method in the above embodiment.

[0125] The memory 22 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes can also be transient storage media.

[0126] In addition, the specific processes of loading and executing multiple instructions by the above storage medium and the terminal device have been described in detail in the above method and will not be repeated here.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target detection method, characterized in that, The target detection method includes: Obtain multi-scale image features of the image to be detected and text features of a preset prompt text; Interact the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features; Based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features other than the high-level semantic image features, determine multi-modal fusion features; Based on the multi-modal fusion features, determine the detection information of the image to be detected; Among them, the determining of the multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features other than the high-level semantic image features specifically includes: Use the image feature with the smallest resolution among the updated high-level semantic image features and the low-level semantic image features as the first target feature, and the text feature as the second target feature for feature fusion to obtain a first target fusion feature; Use the image feature with the second smallest resolution among the updated high-level semantic image features and the low-level semantic image features as the first target feature, and the first target fusion feature as the second target feature for feature fusion to obtain a second target fusion feature; And so on until the Nth target fusion feature is obtained, where N is the number of features of the multi-scale image features; Stitch the first target fusion feature to the Nth target fusion feature to obtain multi-modal fusion features; The fusion process of the feature fusion specifically includes: Determine a stitched feature based on the first target feature and the second target feature; Construct a first intermediate feature and a second intermediate feature based on the stitched feature, and determine a third intermediate feature based on the first intermediate feature; Add and flatten the second intermediate feature and the third intermediate feature to fuse the first target feature and the second target feature.

2. The object detection method according to claim 1, wherein, Before the interacting of the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features, the method further includes: Obtain the image resolution of each scale of image features; Divide the multi-scale image features into high-level semantic image features and low-level semantic image features based on the image resolution, where the resolution of any high-level semantic image feature is less than that of any low-level semantic image feature.

3. The object detection method according to claim 1 or 2, characterized in that, The interacting of the high-level semantic image features in the multi-scale image features with the text features to obtain updated high-level semantic image features and updated text features specifically includes: Construct a query vector based on the high-level semantic image features, and perform interactive learning on the high-level semantic image features and the text features to obtain updated high-level semantic image features; Use the text feature as the query vector, and perform interactive learning on the text feature and the updated high-level semantic image features to obtain updated text features.

4. The object detection method according to claim 3, characterized in that Constructing a query vector based on the high-level semantic image features, and performing interactive learning on the high-level semantic image features and the text features to obtain the updated high-level semantic image features specifically includes: Performing self-attention learning on the high-level semantic image features to obtain intermediate high-level semantic image features; Constructing a query vector with the intermediate high-level semantic image features, constructing value vectors and key vectors with the text features, and performing cross-attention learning based on the query vector, value vectors, and key vectors to obtain the updated high-level semantic image features.

5. The object detection method according to claim 1 or 2, characterized in that, The interaction between the high-level semantic image features in the multi-scale image features and the text features to obtain the updated high-level semantic image features and the updated text features specifically includes: Using the text features as query vectors, and performing interactive learning on the text features and the high-level semantic image features to obtain the updated text features; Constructing a query vector based on the high-level semantic image features, and performing interactive learning on the high-level semantic image features and the updated text features to obtain the updated high-level semantic image features.

6. The object detection method according to claim 1, wherein The determination of the third intermediate feature based on the first intermediate feature is specifically: Inputting the first intermediate feature into a reparameterization module, and outputting the third intermediate feature through the reparameterization module, where the network parameters of the reparameterization module integrate the parameters of the convolutional layer and the normalization layer, and the convolutional calculation and normalization operations are realized through one-step calculation.

7. The object detection method according to claim 1, wherein The determination of the concatenated feature based on the first target feature and the second target feature specifically includes: For the case where the image feature with the smallest resolution is used as the first target feature, concatenating the first target feature and the second target feature to obtain the concatenated feature; For cases other than using the image feature with the smallest resolution as the first target feature, performing upsampling on the second target feature to obtain an upsampled image feature, and concatenating the upsampled image feature and the first target feature to obtain the concatenated feature, where the image size of the upsampled image feature is the same as the feature size of the first target feature.

8. A target detection device, characterized in that, The target detection device specifically includes: A feature extraction module, configured to obtain multi-scale image features of the image to be detected and text features of a preset prompt text; A multi-modal feature fusion module, configured to interact the high-level semantic image features in the multi-scale image features with the text features to obtain the updated high-level semantic image features and the updated text features, and determine multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features other than the high-level semantic image features; A decoding module, configured to determine the detection information of the image to be detected based on the multi-modal fusion features; Among them, the determination of the multi-modal fusion features based on the updated high-level semantic image features, the updated text features, and the low-level semantic image features in the multi-scale image features other than the high-level semantic image features specifically includes: Using the image feature with the smallest resolution among the updated high-level semantic image features and low-level semantic image features as the first target feature, and the text feature as the second target feature for feature fusion to obtain the first target fusion feature; Using the image feature with the second smallest resolution among the updated high-level semantic image features and low-level semantic image features as the first target feature, and the first target fusion feature as the second target feature for feature fusion to obtain the second target fusion feature; And so on until the Nth target fusion feature is obtained, where N is the number of features of the multi-scale image features; Concatenating the first target fusion feature to the Nth target fusion feature to obtain the multi-modal fusion feature; The fusion process of the feature fusion specifically includes: Determining the concatenated feature based on the first target feature and the second target feature; Constructing a first intermediate feature and a second intermediate feature based on the concatenated feature, and determining a third intermediate feature based on the first intermediate feature; Adding and flattening the second intermediate feature and the third intermediate feature to fuse the first target feature and the second target feature.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the object detection method according to any one of claims 1-7.

10. A terminal device, characterized in that, Including: A processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the object detection method according to any one of claims 1-7.

Images