An open vocabulary target detection method, system, device and medium

By using the Mamba module and the multi-head attention Transformer module for image text fusion in open vocabulary object detection, the problem of large amount of computing and poor real-time performance caused by the Transformer architecture is solved, and more efficient object detection accuracy and performance are achieved.

CN118673465BActive Publication Date: 2024-11-05QINGDAO UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411155638.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-05
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The existing open vocabulary object detection method based on Transformer architecture is large in computing, resulting in poor real-time performance of object detection.

Method used

The Mamba module is used to fusion of image text, and the Transformer module with the multi-head attention mechanism is used to fusion of images and text. The bounding box module is used to obtain target detection results, reducing the calculation amount and improving accuracy.

Benefits of technology

It effectively improves the accuracy and computing efficiency of image text fusion, improves the accuracy and performance of object detection, and ensures the comprehensive and accurate representation of multimodal data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673465B_ABST
    Figure CN118673465B_ABST
Patent Text Reader

Abstract

The present invention provides an open vocabulary target detection method, system, device and medium, which belongs to the technical field of target detection. The scheme introduces Mamba into the target detection task, replaces the traditional Transformer with Mamba to fuse the image and text, and adopts the image-text fusion module based on Mamba to effectively improve the accuracy of image-text fusion. At the same time, it can effectively reduce the amount of calculation in the image-text fusion process, improve the calculation efficiency, and thus effectively improve the target detection accuracy; on the basis of the image-text fusion based on Mamba, the scheme introduces a multi-head attention Transformer module to enhance the alignment between image and text features, ensuring a more comprehensive and accurate representation of multimodal data; by adopting the multi-head attention mechanism, the module effectively captures complex dependencies and improves the overall performance of detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to an open vocabulary target detection method, system, device and medium. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Different from the traditional model of closed-domain object detection, open-vocabulary object detection aims to recognize objects from a wide and potentially unlimited set of categories, requiring the model to understand and generalize a variety of visual and textual information;

[0004] Currently, existing open vocabulary object detection methods usually use the Transformer architecture for image-text fusion. Although the Transformer shows certain effects in the image-text fusion process, the object detection methods based on the Transformer architecture often have a large amount of computation, resulting in poor real-time performance of object detection. Summary of the invention

[0005] The embodiments of the present invention provide an open vocabulary target detection method, system, device and medium to solve the problem that the target detection method based on the Transformer architecture in the prior art has a large amount of calculation, resulting in poor real-time performance of the target detection.

[0006] According to a first aspect of an embodiment of the present invention, there is provided an open vocabulary object detection method, comprising:

[0007] Get the image to be detected and its text description;

[0008] Based on the image and its text description, using a pre-trained deep learning-based object detection model to obtain an object detection result;

[0009] Among them, the target detection model based on deep learning specifically performs the following processing procedures: extracting image features and text features of different scales respectively; based on the image features and text features of each scale, using the corresponding Mamba-based image-text fusion module to perform image-text fusion, and obtaining the corresponding image-text fusion features at different scales; using the image-text fusion features at different scales as the input of the target detection head to obtain the target detection result of the image.

[0010] Furthermore, the target detection head specifically performs the following process: based on the image-text fusion features and text features at different scales, a neural network model based on a multi-head attention mechanism is used to perform secondary fusion of the image and text, and the image and text features are aligned by calculating the similarity between the secondary fusion results and the text features; based on the aligned image and text features, the target detection result is obtained through a bounding box module.

[0011] Furthermore, the deep learning-based target detection model includes an image encoder for image feature extraction, a text encoder for text feature extraction, an image-text fusion module for image and text feature fusion, and a target detection head for target detection, wherein the fusion of image features and text features of different scales corresponds to different image-text fusion modules.

[0012] Furthermore, the image-text fusion module specifically performs the following processing process: for image features and text features of any scale of input, two-dimensional convolution, Flatten and Transpose processing are sequentially performed on the image features, and the processed image features and text features are spliced; the spliced ​​features are used as the input of the Mamba module, and the output results of the Mamba module are sequentially processed with one-dimensional convolution and two-dimensional convolution to obtain image-text fusion features.

[0013] Furthermore, the secondary fusion of the image and text is specifically as follows: taking the image-text fusion features and the text features as input, and utilizing the Transformer module based on the multi-head attention mechanism to further fuse the image and text.

[0014] Furthermore, the Transformer module based on the multi-head attention mechanism specifically performs the following processing procedures: Reshape and Linear operations are sequentially performed on the input image-text fusion features to obtain a query vector; the first Linear and second Linear operations are respectively performed on the input text features to obtain a key vector and a value vector; attention features are obtained based on the dot product of the query vector and the key vector processed by the Reshape and Transpose operations; the attention features are normalized, and after dot multiplication with the value vector, they are processed sequentially by Transpose, Linear, Reshape and Transpose.

[0015] Furthermore, the acquisition of the target detection result is specifically as follows: based on the obtained secondary fusion features and text features, similarity calculation is performed using a pre-built image-text similarity calculation module; based on the similarity calculation result, target detection in the image is achieved through a bounding box module; the image-text similarity calculation module uses a CNN network, which takes the spliced ​​secondary fusion features and text features as input, and sequentially undergoes two-dimensional convolution and activation function to obtain a similarity calculation result.

[0016] According to a second aspect of an embodiment of the present invention, there is provided an open vocabulary object detection system, comprising:

[0017] A data acquisition unit, which is used to acquire an image to be detected and a text description thereof;

[0018] An object detection unit, configured to obtain an object detection result based on the image and its text description using a pre-trained deep learning-based object detection model;

[0019] Among them, the target detection model based on deep learning specifically performs the following processing procedures: extracting image features and text features of different scales respectively; based on the image features and text features of each scale, using the corresponding Mamba-based image-text fusion module to perform image-text fusion, and obtaining the corresponding image-text fusion features at different scales; using the image-text fusion features at different scales as the input of the target detection head to obtain the target detection result of the image.

[0020] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored and running on the memory, wherein the processor implements the open vocabulary object detection method when executing the program.

[0021] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the open vocabulary object detection method when executed by a processor.

[0022] One or more of the above technical solutions have the following beneficial effects:

[0023] The present invention provides an open vocabulary target detection method, system, device and medium. The scheme introduces Mamba into the target detection task and replaces the traditional Transformer with Mamba to fuse the image and text. On the one hand, the proposed image-text fusion module based on Mamba can effectively improve the accuracy of image-text fusion, and on the other hand, it can effectively reduce the amount of calculation in the image-text fusion process, improve the calculation efficiency, and thus effectively improve the target detection accuracy. On the basis of the image-text fusion based on Mamba, the present invention introduces a multi-head attention Transformer module to enhance the alignment between image and text features, ensuring a more comprehensive and accurate representation of multimodal data. By adopting a multi-head attention mechanism, the module effectively captures complex dependencies and improves the overall performance of the detection scheme. The advantages of the additional aspects of the present invention will be partially given in the following description, and part will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 A flow chart of an open vocabulary object detection method according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the image-text fusion layer structure according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the Transformer module structure based on the multi-head attention mechanism described in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0030] Terminology explanation:

[0031] Flatten: Flattening operation in deep learning;

[0032] Transpose: Transpose operation in deep learning;

[0033] Concat: concatenation operation in deep learning;

[0034] Linear: Linear processing operations in deep learning;

[0035] Reshape: used to change the shape of data in deep learning;

[0036] Bn: Batch Normalization.

[0037] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0038] like Figure 1 As shown, the embodiment of the present invention provides an open vocabulary object detection method, which specifically includes the following processing procedures:

[0039] Step 1: Get the image to be detected and its text description;

[0040] Step 2: Based on the image and its text description, a pre-trained deep learning-based object detection model is used to obtain an object detection result;

[0041] Among them, the target detection model based on deep learning specifically performs the following processing procedures: extracting image features and text features of different scales respectively; based on the image features and text features of each scale, using the corresponding Mamba-based image-text fusion module to perform image-text fusion, and obtaining the corresponding image-text fusion features at different scales; using the image-text fusion features at different scales as the input of the target detection head to obtain the target detection result of the image.

[0042] In a specific implementation, the target detection head specifically performs the following process: based on the image-text fusion features and text features at different scales, a neural network model based on a multi-head attention mechanism is used to perform secondary fusion of the image and text, and the image and text features are aligned by calculating the similarity between the secondary fusion results and the text features; based on the aligned image and text features, the target detection result is obtained through a bounding box module.

[0043] In a specific implementation, the deep learning-based target detection model includes an image encoder for image feature extraction, a text encoder for text feature extraction, an image-text fusion module for image and text feature fusion, and a target detection head for target detection, wherein the fusion of image features and text features of different scales corresponds to different image-text fusion modules.

[0044] In a specific implementation, the image feature extraction is based on the backbone module in YOLO-World, and the feature extraction of different scales of the image is realized through the backbone module; the text feature extraction is based on the text feature extraction module in the multimodal pre-training model.

[0045] Specifically, the image and text are input to the image encoder and text encoder respectively to extract image and text features. The image encoder outputs multi-scale image features, expressed as ( I 1, I 2, ..., I i , ..., I n ),in, I i is the i-th image feature. In this embodiment, n Set to 3; the dimension of each feature map is C × H × W , C ×( H / 2)×( W / 2), ..., C ×( H / n )×( W / n )}.in, C is the channel of the image feature, H and W are the height and width of the image features respectively. The feature map size of the text after passing through the text encoder is K × M ,in, K is the number of categories of text features, M is the channel of text features. The text features are input into the corresponding Mamba-based image-text fusion layer to achieve the fusion of image features and text features. Finally, the fused features are input into the target detection head to complete target detection.

[0046] In a specific implementation, the image-text fusion module specifically performs the following processing process: for image features and text features of any scale of input, the image features are sequentially subjected to two-dimensional convolution, flattening and transposition processing, and the processed image features and text features are spliced; the spliced ​​features are used as the input of the Mamba module, and the output results of the Mamba module are sequentially subjected to one-dimensional convolution and two-dimensional convolution processing to obtain image-text fusion features.

[0047] Specifically, since images provide visual information and texts provide semantic information, after extracting their respective features, the image and text features are integrated by using a multimodal data fusion module (i.e., image-text fusion module). This complementary information can effectively enhance the accuracy of open vocabulary object detection.

[0048] like Figure 2 As shown in Figure 1, the image-text fusion module based on the Mamba module is mainly composed of a two-dimensional convolution block, a mamba block, and a one-dimensional convolution block. Specifically, the image features I i First, it is resized to an appropriate size through a 2D convolution block (including 2D convolution and regularization operations), and then it is flattened and transposed in sequence to make its feature dimension the same as the text feature dimension. It is then concatenated with the text feature and sent to the Mamba module (consisting of N Mamba layers), and then it is sequentially subjected to 1D convolution and 2D convolution operations to obtain the fused feature. F i .

[0049] In a specific implementation, the secondary fusion of the image and text is specifically: taking the image-text fusion features and text features as input, and utilizing the Transformer module based on the multi-head attention mechanism to further fuse the image and text.

[0050] In a specific implementation, the Transformer module based on the multi-head attention mechanism specifically performs the following processing procedures: Reshape and Linear operations are sequentially performed on the input image-text fusion features to obtain a query vector; the first Linear and second Linear operations are respectively performed on the input text features to obtain a key vector and a value vector; attention features are obtained based on the dot product of the query vector and the key vector processed by the Reshape and Transpose operations; the attention features are normalized, and after dot multiplication with the value vector, they are sequentially processed by Transpose, Linear, Reshape and Transpose.

[0051] Specifically, Figure 3 As shown in the figure, the target detection head performs the following processing steps:

[0052] In order to eliminate the differences in scale, format and semantic structure between image and text features, the image and text features are aligned through the Transformer-based text-image similarity module after the fusion process. This module achieves effective alignment of multimodal data by mapping image and text features into a unified space. The object detection head contains two main components: the Transformer-based text-image similarity module and the bounding box module. The Transformer-based text-image similarity module aligns the two types of features by calculating the similarity between the image and text. It can be further subdivided into the image-text fusion multi-head attention Transformer module and the similarity module.

[0053] The object detection head consists of two modules: a Transformer-based text-image similarity module and a Boundingbox module.

[0054] The Transformer-based text-image similarity module includes: a Transformer module based on a multi-head attention mechanism and a similarity calculation module; wherein the Transformer module based on a multi-head attention mechanism is used to further integrate the image and text. It takes the integrated image-text features and text features as input, firstly integrates the integrated features F i Reshape and input the linear layer to obtain Query (Q: query vector), and send the text feature T to two linear layers to obtain Key (K: key vector) and Value (V: value vector). Then reshape and transpose K, multiply the result with Q to get the attention feature, and then perform Softmax operation on it to normalize the attention, then dot multiply it with V, and perform Transpose and linear operation on the result, and finally reshape and transpose the result to make it (that is, ) has the same dimension as the text feature T.

[0055] After the fusion of the image and text is completed, before the bounding box module is sent, this embodiment sets up an image-text similarity calculation module. Different from the similarity module of YOLO-World, the image-text similarity of this embodiment is realized through a CNN network. Specifically, first Concat operation is performed with T, and then it is sent to the two-dimensional convolution block and activation function in sequence. The activation function used in this embodiment is ReLU;

[0056] Since YOLO-World implements the similarity module through a function, it can only capture simple similarity relationships. The solution described in this embodiment is implemented through a CNN layer. Compared with YOLO-World, it can capture more complex features and nonlinear relationships, and improve the accuracy and robustness of similarity calculation.

[0057] Furthermore, the bounding box module uses the secondary fusion features from the image-text fusion multi-head attention Transformer module Accurately locate and classify objects in the image. The bounding box module uses the same structure as YOLO-World, so it will not be repeated here.

[0058] The effectiveness of the detection head relies on the Transformer-based text-image similarity module, which integrates visual and textual data into a powerful representation of objects. This comprehensive integration is crucial for the model's enhanced perception in complex, multimodal scenes, enabling it to interpret and respond to various visual contexts with high precision. Through the above features, the scheme described in this embodiment can not only accurately locate and classify objects, but also have a detailed understanding of their contextual associations in the scene, which is crucial for applications in dynamic environments (such as autonomous navigation and interactive robots).

[0059] In the specific implementation, the solution described in this embodiment adopts the same loss function as YOLO-World, so it will not be repeated here.

[0060] In one or more embodiments, the present invention provides an open vocabulary object detection system corresponding to the above-mentioned open vocabulary object detection method, including:

[0061] A data acquisition unit, which is used to acquire an image to be detected and a text description thereof;

[0062] An object detection unit, configured to obtain an object detection result based on the image and its text description using a pre-trained deep learning-based object detection model;

[0063] Among them, the target detection model based on deep learning specifically performs the following processing procedures: extracting image features and text features of different scales respectively; based on the image features and text features of each scale, using the corresponding Mamba-based image-text fusion module to perform image-text fusion, and obtaining the corresponding image-text fusion features at different scales; using the image-text fusion features at different scales as the input of the target detection head to obtain the target detection result of the image.

[0064] In a specific implementation, the target detection head specifically performs the following process: based on the image-text fusion features and text features at different scales, a neural network model based on a multi-head attention mechanism is used to perform secondary fusion of the image and text, and the image and text features are aligned by calculating the similarity between the secondary fusion results and the text features; based on the aligned image and text features, the target detection result is obtained through a bounding box module.

[0065] In further embodiments, there is also provided:

[0066] An electronic device comprises a memory, a processor and a computer program stored and running on the memory, wherein the processor implements the open vocabulary target detection method when executing the program.

[0067] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the open vocabulary object detection method.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An open vocabulary object detection method, It is characterized in that include: Get the image to be detected and its text description; Based on the image and its text description, using a pre-trained deep learning-based object detection model to obtain an object detection result; The target detection model based on deep learning specifically performs the following processing: extracting image features and text features of different scales respectively; based on the image features and text features of each scale, using the corresponding Mamba-based image-text fusion module to perform image-text fusion, and obtaining the corresponding image-text fusion features at different scales; using the image-text fusion features at different scales as the input of the target detection head, and obtaining the target detection result of the image; The target detection head specifically performs the following process: based on the image-text fusion features and text features at different scales, a neural network model based on a multi-head attention mechanism is used to perform secondary fusion of the image and text, and the image and text features are aligned by calculating the similarity between the secondary fusion result and the text features; based on the aligned image and text features, a target detection result is obtained through a bounding box module; The image-text fusion module specifically performs the following processing: for image features and text features of any scale input, two-dimensional convolution, Flatten and Transpose processing are sequentially performed on the image features, and the processed image features and text features are spliced; the spliced ​​features are used as the input of the Mamba module, and the output results of the Mamba module are sequentially processed with one-dimensional convolution and two-dimensional convolution to obtain image-text fusion features; The secondary fusion of the image and text is specifically as follows: taking the image-text fusion feature and the text feature as input, and using the Transformer module based on the multi-head attention mechanism to further fuse the image and text; The Transformer module based on the multi-head attention mechanism specifically performs the following processing procedures: Reshape and Linear operations are sequentially performed on the input image-text fusion features to obtain a query vector; the first Linear and second Linear operations are respectively performed on the input text features to obtain a key vector and a value vector; attention features are obtained based on the dot product of the query vector and the key vector processed by the Reshape and Transpose operations; the attention features are normalized, and after dot multiplication with the value vector, they are processed sequentially by Transpose, Linear, Reshape and Transpose.

2. An open vocabulary object detection method as claimed in claim 1, It is characterized in that The deep learning-based target detection model includes an image encoder for image feature extraction, a text encoder for text feature extraction, an image-text fusion module for image and text feature fusion, and a target detection head for target detection, wherein the fusion of image features and text features of different scales corresponds to different image-text fusion modules.

3. The open vocabulary object detection method according to claim 1, It is characterized in that The acquisition of the target detection result is specifically as follows: based on the obtained secondary fusion features and text features, a pre-built image-text similarity calculation module is used to perform similarity calculation; based on the similarity calculation result, target detection in the image is achieved through a bounding box module; the image-text similarity calculation module uses a CNN network, which takes the spliced ​​secondary fusion features and text features as input, and sequentially obtains the similarity calculation result through two-dimensional convolution and activation function.

4. An open vocabulary object detection system, It is characterized in that include: A data acquisition unit, which is used to acquire an image to be detected and a text description thereof; An object detection unit, configured to obtain an object detection result based on the image and its text description using a pre-trained deep learning-based object detection model; Among them, the target detection model based on deep learning specifically performs the following processing process: extract image features and text features of different scales respectively; based on the image features and text features of each scale, use the corresponding Mamba-based image-text fusion module to perform image-text fusion, and obtain the corresponding image-text fusion features at different scales; use the image-text fusion features at different scales as the input of the target detection head to obtain the target detection result of the image; the target detection head specifically performs the following process: based on the image-text fusion features and text features at different scales, use the neural network model based on the multi-head attention mechanism to perform secondary fusion of image and text, and align the image and text features by calculating the similarity between the secondary fusion result and the text feature; based on the aligned image and text features, obtain the target detection result through the bounding box module; The target detection head specifically performs the following process: based on the image-text fusion features and text features at different scales, a neural network model based on a multi-head attention mechanism is used to perform secondary fusion of the image and text, and the image and text features are aligned by calculating the similarity between the secondary fusion result and the text features; based on the aligned image and text features, a target detection result is obtained through a bounding box module; The image-text fusion module specifically performs the following processing: for image features and text features of any scale input, two-dimensional convolution, Flatten and Transpose processing are sequentially performed on the image features, and the processed image features and text features are spliced; the spliced ​​features are used as the input of the Mamba module, and the output results of the Mamba module are sequentially processed with one-dimensional convolution and two-dimensional convolution to obtain image-text fusion features; The secondary fusion of the image and text is specifically as follows: taking the image-text fusion feature and the text feature as input, and using the Transformer module based on the multi-head attention mechanism to further fuse the image and text; The Transformer module based on the multi-head attention mechanism specifically performs the following processing procedures: Reshape and Linear operations are sequentially performed on the input image-text fusion features to obtain a query vector; the first Linear and second Linear operations are respectively performed on the input text features to obtain a key vector and a value vector; attention features are obtained based on the dot product of the query vector and the key vector processed by the Reshape and Transpose operations; the attention features are normalized, and after dot multiplication with the value vector, they are processed sequentially by Transpose, Linear, Reshape and Transpose.

5. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, It is characterized in that When the processor executes the program, an open vocabulary object detection method as described in any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, an open vocabulary object detection method as described in any one of claims 1 to 3 is implemented.