Visual language target detector Zone-YOLO based on area prompt
By introducing Scale-Aware VL-PAN Neck and Adapter modules in visual language object detection, combined with Text Contrastive Head and Zone Head, the problems of feature fusion between modals and text information utilization are solved, achieving more efficient detection effects, especially It is the improvement of AP indicators on the COCO dataset.
Patent Information
- Application Number
- CN202411048379.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The existing visual language object detection methods have inconsistent characteristics fusion between modals and the utilization of text information, which leads to the intensification of the gap between modals. Most methods ignore the effect of text features on target box regression, resulting in detection. Limited effect.
A visual language object detector based on area cues was proposed. The Scale-Aware VL-PAN Neck module was used to perform multimodal multi-scale feature alignment and fusion, and the comparison and learning was performed through Text Contrastive Head, and the categories and regions were captured using Adapter. Co-occurrence information and guide the detector to learn the triple co-occurrence information of the zone-category-entity in the Zone Head.
The seamless fusion of multimodal representations at different scales was achieved, which improved the detection effect, especially the AP indicator on the COCO benchmark data set reached 55.1, which was significantly better than the baseline model and achieved competitive results.
Smart Images

Figure CN118940803A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to Zone-YOLO, which is a visual language target detection implementation scheme based on region prompts. Background Art
[0002] Object detection is an important task in machine vision. It is used to identify and locate the region of interest (ROI) in an image. It is also the basis for many tasks such as image understanding and visual positioning. In the past few decades, work on object detection has made significant breakthroughs. These methods can be roughly divided into two-stage detectors represented by Faster RCNN, and single-stage detectors represented by the DETR series and YOLO series. The former divides object detection into two subtasks: regional hints and hint optimization, but the reasoning speed is slow. The latter integrates the detection process into one stage, uses an end-to-end architecture and a lightweight backbone, performs well in real-time object detection, and is widely used in fields such as intelligent transportation, defect detection, and remote sensing technology.
[0003] Vision-Language Models (VLMs) can obtain more general and robust feature representations by fusing multimodal information. The Vision-Language Model-Based Object Detection (VLOD) method obtains more semantic information from the text through a text encoder, alleviating the semantic deficiency problem of single visual modality methods, thereby significantly improving the detection effect.
[0004] There are two issues worth noting in VLOD research:
[0005] Problem 1) The inconsistency between modalities is inherent. Therefore, the efficient fusion of visual and language features is the key to whether the model can fully utilize feature information and enhance its representation ability. For the first problem, existing methods ignore the macro and micro concepts that distinguish text features, and indiscriminately fuse these concepts with visual features of different scales, which will disrupt image features and worsen the modality gap.
[0006] Problem 2) Most VLOD methods focus on using text features for contrastive learning in the classification branch, without fully exploring the role of text information in the regression branch.
[0007] Conventional VLOD methods ignore the use of language features for target box regression:
[0008] (i) The problem of difficulty in describing reference exists, and the random matching method causes the detector to only learn the supervision signals of part of the positive samples each time.
[0009] (ii) Object detection cannot know the object to be detected in advance, so the category-related region hints cannot be directly used by the present invention. Summary of the invention
[0010] In order to overcome the shortcomings of the prior art, the present invention discloses a visual language object detector Zone-YOLO based on region cue.
[0011] Technical solution:
[0012] A visual language target detector Zone-YOLO based on regional prompts includes a visual encoder, a text encoder, a Scale-Aware VL-PAN Neck, a word embedding, a zone prompt, an Adapter, a Text Contrastive Head, and a Zone Head, wherein:
[0013] The visual encoder uses the pre-trained YOLOv8 Backbone, and the output image is encoded into multi-scale image features;
[0014] The text encoder uses CLIP text encoder, inputs class names and zone phrases, and encodes output as word embeddings and zone prompts;
[0015] The Scale-Aware VL-PAN Neck takes the multi-scale image features output by the visual encoder and the word embedding output by the text encoder as input to achieve alignment and fusion of multi-modal and multi-scale features;
[0016] The Text Contrastive Head takes the output of Scale-Aware VL-PAN Neck and word embedding as input, calculates the similarity between image features and text features, and performs comparative learning;
[0017] The Adapter takes the output word embedding and zone prompts of the text encoder as input, captures the co-occurrence information of categories and zones, and obtains class-specific zone embeddings;
[0018] The Zone Head,takes the output of Scale-Aware VL-PAN Neck and Adapter as input,captures the triple co-occurrence of region-category-entity,and then guides the detector for better regression.
[0019] The visual language target detector Zone-YOLO implemented by the present invention with a YOLO structure has the following main innovations and beneficial effects:
[0020] 1. This paper pioneered a scale-aware multimodal fusion module, which fully exploits text features and learns to seamlessly fuse multimodal representations at different scales.
[0021] 2. The present invention proposes a new region hint learning module, which introduces text features into the regression head and captures the region-category-instance triple co-occurrence information.
[0022] 3. Fine-tuning on the COCO benchmark dataset proves the superior performance of the Zone-YOLO proposed in this invention. Compared with the baseline model, the Zone-YOLO of this invention has been significantly improved in various indicators, among which the AP indicator has been improved by 1.7 to 55.1. Compared with other similar algorithms, the Zone-YOLO of this invention has achieved competitive results. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 Schematic diagram of the overall structure of Zone-YOLO of the present invention.
[0024] Figure 2 Schematic diagram of multimodal attention mechanism. (a) is the dual-stream multimodal attention mechanism in the channel direction of the channel-enhanced SAMF submodule, and (b) is the dual-stream multimodal attention mechanism in the channel direction of the modality-enhanced SAMF submodule. In the figure, I represents image features, T represents text features, and FFN (feedforward neural network) shares parameters for the two modalities and introduces a learnable parameter θ to control the skip layer connection.
[0025] Figure 3 The following is a schematic diagram of the adapter used to obtain category-related regional embeddings, where P stands for regional prompts, W stands for word embeddings, and LSA stands for Language Semantic Attention.
[0026] Figure 4 Schematic diagram of the self-attention module. (a) is the self-attention module for MI, and (b) is the aggregation of MI into image features. In the figure, I represents image features and Z represents region embedding. DETAILED DESCRIPTION
[0027] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of the present application will become more apparent with the following description.
[0028] Model structure
[0029] The overall structure of Zone-YOLO is as follows Figure 1 As shown:
[0030] A visual language object detector Zone-YOLO based on regional prompts includes a visual encoder, a text encoder, a Scale-Aware VL-PAN Neck, a word embedding, zone prompts, an Adapter, a Text Contrastive Head, and a Zone Head, wherein:
[0031] The visual encoder uses the pre-trained YOLOv8 Backbone, and the output image is encoded as multi-scale image features.
[0032] The text encoder uses CLIP text encoder, inputs class names and zone phrases, and encodes output as word embeddings and zone prompts.
[0033] The Scale-Aware VL-PAN Neck is used to achieve alignment and fusion of multi-modal and multi-scale features, which is one of the key innovations of the present invention.
[0034] The Scale-Aware VL-PAN Neck includes a scale-aware modal fusion (SAMF) module, a channel-enhanced SAMF submodule, and a modality-enhanced SAMF submodule.
[0035] The scale-aware modal fusion (SAMF) module further comprises a plurality of modal mixing operation SAMF submodules connected in series;
[0036] The channel-enhanced SAMF submodule and the modality-enhanced SAMF submodule each include a corresponding multimodal attention mechanism;
[0037] in:
[0038] Firstly, the scale-aware modal mixing feature matrix (MI) is obtained through the modal mixing operation of the scale-aware modal fusion (SAMF) module, and the features are supplemented and interacted to achieve the alignment and fusion of visual and language features.
[0039] Afterwards, based on the corresponding multimodal attention mechanisms of the channel-enhanced SAMF submodule and the modality-enhanced SAMF submodule, the modality mixture feature matrix (MI) is used to enhance the image and text features.
[0040] The modal mixing operation SAMF submodule:
[0041] First, the image and text features are projected into the same feature space (the number of channels of the two are aligned by default) to obtain the coarse-grained "modal mixing feature matrix MI" modal fusion feature matrix Where HW represents the spatial dimension of the image, N represents the category dimension, and C represents the number of channels.
[0042] Next, scale-aware query As a mask, it is loaded onto the modal fusion feature matrix MI, and the scale-aware query SQ is used to activate the corresponding scale signal and suppress the signals of other scales, thereby obtaining the scale-aware modal mixing feature. The whole process is described by formula (1):
[0043] MI=Conv(reshape(SQ)·(W×I))(1)
[0044] Conv is used to adjust the number of feature channels, and regularization and affine operations are omitted. The scale-aware query SQ can be a learnable parameter matrix or the spatial attention weight of the image. H represents the height of the image, and W represents the width of the image.
[0045] The modal fusion feature matrix MI after scale-aware modal mixing is a coarse fusion feature.
[0046] The channel-enhanced SAMF submodule and the modality-enhanced SAMF submodule use two multimodal attention mechanisms combined with a scale-aware modality fusion feature matrix MI to perform feature interaction and enhancement in the channel and modality dimensions, respectively.
[0047] The channel-enhanced SAMF submodule is a dual-stream multimodal attention mechanism in the channel direction, which gradually refines and fuses the modal fusion feature matrix MI into the text stream and the image stream, thereby achieving adaptive feature enhancement. Figure 2 (a). More specifically, we first perform a channel attention-like operation on the modality fusion feature matrix MI, compress the spatial and category dimensions using global average pooling (GAP), and obtain an attention vector of length C through FFN (feedforward neural network) and Sigmoid as the channel-wise global distribution response. Then, we multiply the attention vector element by element with the image and text features to complete the attention weighting. This is expressed by formula (2):
[0048]
[0049] Among them, θ 1 ,θ 2 Used to control skip-layer connections, I represents image features, and T represents text features.
[0050] The modality-enhanced SAMF submodule uses the spatial and category dimensions in the modality fusion feature matrix MI to guide the fusion of the other modalities, adds scale-aware text guidance to the image features, and adds scale-aware image guidance to the text features. Figure 2 (b). Specifically, the maximum value is selected in the category direction of the modal fusion feature matrix MI, and then the sigmoid operation is used to obtain the attention weight, which is used to aggregate the text semantic information into the image features. Global average pooling and sigmoid operations are used in the spatial direction of the modal fusion feature matrix MI to aggregate the image feature information with the text features. The above process is expressed by formula (3):
[0051]
[0052] Among them, δ represents the sigmoid operation, θ 1 ,θ 2 Used to control skip-layer connections, I represents image features, T represents text features, N represents category dimensions, and HW represents the spatial dimension of the image.
[0053] The Text Contrastive Head is an existing method “YOLO-World: Real-Time Open-Vocabulary Object Detection” [1] The module structure in Scale-Aware VL-PAN Neck takes the output and word embeddings of Scale-Aware VL-PAN Neck as input, calculates the similarity between image features and text features, and performs comparative learning.
[0054] The relationship between the various components:
[0055] The output of zone prompts is connected to Adapter, and then connected to Zone Head through the output of Adapter. Zone Head is used to capture the triple co-occurrence of zone, category and entity. Zone prompts explicitly introduce location information into Zone Head to guide the regression of target box. Category-independent zone prompts are used to solve the problem of reference difficulty. Adapter is proposed to obtain category-related zone embeddings to capture category-region co-occurrence; Zone Head obtains region-category-entity co-occurrence features; finally, the self-supervised auxiliary branch is used to enhance the stability of zone embeddings.
[0056] The adapter is used to capture the category-region co-occurrence information of word embedding and zone prompts. The adapter captures the co-occurrence information of categories and regions to obtain "class-specific zone embedding".
[0057] Zone Head fuses the zone embeddings output by the Adapter with the output of Scale-Aware VL-PAN Neck to guide the detector to better regress.
[0058] The adapter is composed of a single-layer Transformer module and a Language Sematic Attention (LSA) module. Figure 3 shown.
[0059] zone prompts, including the class-agnostic zone prompts submodule and the class-agnostic zone embedding submodule;
[0060] The class-agnostic zone prompts submodule divides the image into K equal regions, each of which corresponds to a fixed directional noun as a description, and feeds the K descriptions into a text encoder to obtain class-agnostic zone prompts. Where C is the number of channels, which is the same as T. As an example, but not a limitation, K is set to 9, and the 9 regions are upper left, top center, upper right, middle left, center, middle right, lower left, bottom center, and lower right. Because the region hints obtained using text descriptions contain more semantic information, numbers are not used as region descriptions like other methods.
[0061] The category-independent region hint submodule is used to mine the contextual patterns of categories in the image, thereby facilitating object location. In order to fully utilize the semantic representation capabilities of the text encoder and capture the co-occurrence information of categories and regions, the category-independent region hint submodule and the adapter work in the following ways:
[0062] First, the category embedding is used as the value and key matrix, and the region hint is used as the query matrix to input into the Transformer module of the Adapter to capture the long-range semantic relationship between the category and the region. Then, the Language Sematic Attention (LSA) module is used to calculate the mixed matrix of the region hint and category embedding information, and the mixed matrix is weighted by the region information to obtain the category-related region embedding. The whole process is expressed by formula (4):
[0063]
[0064] Wherein δ represents the sigmoid operation; MHSA refers to the multi-head self-attention mechanism (Multi-Head Self-Attention) operation; K is the number of regional prompts P, and the value of the embodiment is 9.
[0065] The above-mentioned mixing matrix weights the regional information, which is a process of multiplying matrix elements. It needs to perform tensor expansion and broadcasting, and the obtained mixing matrix is a coarse-grained information fusion. The Language Sematic Attention (LSA) module is a channel-dimensional max-sigmoid attention, which refines the coarse-grained mixing matrix by constraining the distribution of channel directions, thereby alleviating the problem of feature aliasing. The coarse-grained mixing matrix is used as a query input, and the regional prompts and category embeddings are regarded as two feature inputs. The present invention unifies the structure of the Language Sematic Attention (LSA) module and the scale-aware modal fusion (SAMF) module.
[0066] The Zone Head is used for regression tasks, and its structure is shown in Figure 1 By fusing the entity information of image feature I with the region embedding Z, the Zone Head is used to capture the region-category-entity triple co-occurrence information.
[0067] The Zone Head is a detection head based on regional prompts, including a regression branch detection head, a detection head target box, a self-attention module, an aggregation, and an auxiliary branch. The self-attention module includes "regional dimension self-attention" and "category dimension self-attention;
[0068] The regression branch detection head is used to fuse the input image features with the zone embedding to obtain the region-category-entity co-occurrence features, and the co-occurrence information is interacted through the self-attention module to avoid direct matching of text features and image features, while the auxiliary branch is used to maintain resistance to information loss during the learning process.
[0069] In the regression branch detection head, in the single-stage anchor-free Zone Head, the classification and regression tasks are completely decoupled. Each position of the image feature represents an entity to be regressed, and the channel (usually 4) represents the obtained BBox coordinates.
[0070] The self-attention module:
[0071] Since a new dimension is introduced, the fusion operation needs to trade off the amount of computation. The present invention eliminates the channel when performing matrix multiplication on two tensors to obtain the information mixing matrix The region corresponds to dimension K (the value of K in the embodiment is 9), the category corresponds to dimension N, and the entity corresponds to dimension HW. Losing channel information may aggravate the gap in feature space. Therefore, the present invention designs two self-attention weighted operations of "regional dimension self-attention" and "category dimension self-attention" to interact with information, realize self-attention between entity and category, entity and region, and guide the model to align the feature space of information mixing matrix MI. "Regional dimension self-attention" and "category dimension self-attention" have the same structure. Figure 4 (a) shows the details of the above two operations. Unlike the previous use of the information mixing matrix MI for the enhancement of a single modality, here the information mixing matrix MI is sequentially subjected to max-sigmoid attention. In the region dimension self-attention, the attention weights related to the region and entity are obtained by compressing the category dimension. In the category dimension self-attention, the attention weights related to the category and entity are obtained by compressing the region information. These attention weights are loaded onto the information mixing matrix to enhance the co-occurrence information. The above process is expressed by formula (5) and formula (6):
[0072]
[0073]
[0074] Where j represents the region dimension, i represents the category dimension, δ represents the sigmoid function, and Conv represents 1×1 convolution.
[0075] The aggregation:
[0076] The region-category-entity co-occurrence information obtained through the aggregation operation is entered into the image features to optimize the regression branch of the Zone Head (detection head), such as Figure 4 Specifically, zone embeddings are first used to eliminate the category dimension, and then, referring to formula (3), max-sigmoid attention is used to supplement the regional information into the image features.
[0077] The auxiliary branch provides explicit supervision for zone embeddings. The mean square error (MSE) loss is used to minimize the distance between the enhanced zone embedding Z' and the zone embedding Z, where the enhanced zone embedding It is an enhanced region embedding obtained by eliminating entity dimensions using image features. By aligning the zone embeddings before and after the Zone Head, it alleviates the problem of feature sparsity in the process of mixed information transmission, enhances the stability of features after dimension elimination, and maximizes the regularization effect of the network.
[0078] experiment
[0079] Key results
[0080] Zone-YOLO and the comparison methods are quantitatively analyzed on COCO and LVIS. For fairness, the results of all comparison methods are from publicly published papers or their released codes.
[0081] Comparison results on the COCO dataset.
[0082] Table 1 compares different YOLO object detectors on COCO. We fine-tune Zone-YOLO on COCO train2017 and evaluate it on COCO val2017. For fair comparison, all models use the mask-refine configuration. According to "YOLO-World: Real-Time Open-Vocabulary ObjectDetection" [1] Fine-tune the pre-trained weights on more data. The best results are highlighted in bold.
[0083]
[0084]
[0085] Table 1 shows the performance of Zone-YOLO and recent YOLO family methods. [1][2][3][4][5] Performance on the COCO benchmark. Zone-YOLO and YOLO-World were fine-tuned for 80 epochs based on pre-trained weights, while the other models were trained from scratch and required hundreds of epochs. As can be seen in the table, Zone-YOLO-L has improved across the board compared to the baseline models, especially in AP 75 and AP M, an increase of 2.3 and 1.9 respectively. Compared with “YOLO-World: Real-Time Open-Vocabulary Object Detection” which also uses Vision-Language Neck [1] By comparison, the present invention has achieved comprehensive superiority in all indicators. Thanks to the method proposed in the present invention and large-scale pre-training, Zone-YOLO-L has achieved 75 and AP M The indicators are significantly higher than those of YOLOv9 and YOLOv10, by 1.9 and 2.4, 2.8 and 3.4, and 1.9 and 2.6 respectively. As expected, based on the more generalized fine-tuning of pre-trained weights, Zone-YOLO- In AP,AP 50 ,AP M and AP L The small and medium-sized models also achieved satisfactory results, although there is still a certain gap in a few indicators. Overall, Zone-YOLO achieves competitive results compared to other models in the YOLO series.
[0086] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.
[0087] [1]Cheng T,Song L,Ge Y,et al.YOLO-World:Real-Time Open-VocabularyObject Detection[J].arXiv preprint arXiv:2401.17270,2024.
[0088] [2]Li C, Li L, Jiang H, et al. YOLOv6: A single-stage object detection framework for industrial applications [J]. arXiv preprint arXiv:2209.02976, 2022.
[0089] [3]G.Jocher, A.Chaurasia, and J.Qiu, "Ultralytics YOLO," Jan.2023.[Online].Available: https: / / github.com / ultralytics / ultralytics.
[0090] [4]Wang C Y,Yeh I H,Liao H Y M.YOLOv9:Learning What You Want to LearnUsing Programmable Gradient Information[J].arXiv preprint arXiv:2402.13616,2024.
[0091] [5]Wang A,Chen H,Liu L,et al.YOLOv10:Real-Time End-to-End ObjectDetection[J].arXiv preprint arXiv:2405.14458,2024.
Claims
1. A visual language object detector based on regional cues, Zone-YOLO, It is characterized in that Including visual encoder, text encoder, Scale-Aware VL-PAN Neck, word embedding, zone prompts, Adapter, Text Contrastive Head, Zone Head, among which: The visual encoder uses the pre-trained YOLOv8Backbone, and the output image is encoded as multi-scale image features; The text encoder uses CLIP text encoder, inputs class names and zone phrases, and encodes output as word embedding and zone prompts; The Scale-Aware VL-PAN Neck takes the multi-scale image features output by the visual encoder and the word embedding output by the text encoder as input to achieve alignment and fusion of multi-modal and multi-scale features; The Text Contrastive Head takes the output of Scale-Aware VL-PAN Neck and word embedding as input, calculates the similarity between image features and text features, and performs comparative learning; The Adapter takes the output word embedding and zone prompts of the text encoder as input, captures the co-occurrence information of categories and zones, and obtains class-specific zone embeddings; The Zone Head,takes the output of Scale-Aware VL-PAN Neck and Adapter as input,captures the triple co-occurrence of region-category-entity,and then guides the detector for better regression.
2. A visual language object detector Zone-YOLO based on region cue as claimed in claim 1, It is characterized in that The Scale-Aware VL-PAN Neck includes a scale-aware modal fusion (SAMF) module, a channel-enhanced SAMF submodule, and a modal-enhanced SAMF submodule; the scale-aware modal fusion (SAMF) module further includes a plurality of modal mixing operation SAMF submodules connected in series; the channel-enhanced SAMF submodule and the modal-enhanced SAMF submodule each include a corresponding multimodal attention mechanism; Its working process includes: Firstly, the scale-aware modal mixing feature matrix (MI) is obtained through the modal mixing operation of the scale-aware modal fusion (SAMF) module, and the features are supplemented and interacted to achieve the alignment and fusion of visual and language features. Afterwards, based on the corresponding multimodal attention mechanisms of the channel-enhanced SAMF submodule and the modality-enhanced SAMF submodule, the modality mixture feature matrix (MI) is used to enhance the image and text features.
3. A visual language object detector Zone-YOLO based on region cue as claimed in claim 2, It is characterized in that The modal mixing operation SAMF submodule: First, the image and text features are projected into the same feature space to obtain a coarse-grained "modal mixing feature matrix MI" modal fusion feature matrix Where HW represents the spatial dimension of the image, N represents the category dimension, and C represents the number of channels; Next, scale-aware query As a mask, it is loaded onto the modal fusion feature matrix MI. The scale-aware query SQ is used to activate the corresponding scale signal and suppress the signals of other scales, thereby obtaining the scale-aware modal mixing feature. The whole process is described by formula (1): MI = Conv(reshape(SQ)·(W×I))(1) Conv is used to adjust the number of feature channels; H represents the height of the image; W represents the width of the image.
4. A visual language object detector Zone-YOLO based on region cue as claimed in claim 2, It is characterized in that The channel-enhanced SAMF submodule and the modality-enhanced SAMF submodule use two multimodal attention mechanisms and combine the scale-aware modality fusion feature matrix MI to perform feature interaction and enhancement in the channel and modality dimensions respectively, where: The channel-enhanced SAMF submodule is a dual-stream multimodal attention mechanism in the channel direction, which gradually refines the modal fusion feature matrix MI and fuses it into the text stream and the image stream, thereby realizing adaptive feature enhancement. Specifically, firstly, an operation similar to channel attention is performed on the modal fusion feature matrix MI, and the spatial and category dimensions are compressed using global average pooling (GAP). An attention vector with a length of C is obtained through FFN (feedforward neural network) and Sigmoid as the channel-wise global distribution response; then, the attention vector is element-wise multiplied with the image and text features respectively to complete the attention weighting; it is expressed by formula (2): Among them, θ 1 ,θ 2 Used to control skip-layer connections, I represents image features, and T represents text features; The modality-enhanced SAMF submodule uses the spatial and category dimensions in the modality fusion feature matrix MI to guide the fusion of the other modality, adds scale-aware text guidance to the image features, and adds scale-aware image guidance to the text features. Specifically, the maximum value is selected in the category direction of the modality fusion feature matrix MI, and then the sigmoid operation is used to obtain the attention weight for aggregating the text semantic information into the image features; the global average pooling and sigmoid operation are used in the spatial direction of the modality fusion feature matrix MI to aggregate the image feature information with the text feature. The above process is expressed by formula (3): Among them, δ represents the sigmoid operation, θ 1 ,θ 2 Used to control skip-layer connections, I represents image features, T represents text features, N represents category dimensions, and HW represents the spatial dimension of the image.
5. A visual language object detector Zone-YOLO based on region cue as claimed in claim 1, It is characterized in that The Adapter,is composed of a single-layer Transformer module and a Language Sematic Attention (LSA) module; The zone prompts include a class-agnostic zone prompt submodule and a class-agnostic zone embedding submodule; The class-agnostic zone prompts submodule divides the image into K equal regions, each of which corresponds to a fixed directional noun as a description, and feeds the K descriptions into a text encoder to obtain class-agnostic zone prompts. Where C is the number of channels, which is the same as T; The category-independent region hint submodule is used to mine the contextual patterns of categories in the image, thereby facilitating object location. In order to fully utilize the semantic representation capability of the text encoder and capture the co-occurrence information of categories and regions, the category-independent region hint submodule and the adapter work in a coordinated manner as follows: First, the category embedding is used as the value and key matrix, and the region hint is used as the query matrix to input into the Transformer module of the Adapter to capture the long-range semantic relationship between the category and the region. Then, the Language Sematic Attention (LSA) module is used to calculate the mixed matrix of the region hint and category embedding information, and the mixed matrix is weighted by the region information to obtain the category-related region embedding. The whole process is expressed by formula (4): Where δ represents the sigmoid operation; MHSA refers to the Multi-Head Self-Attention operation; K is the number of regional prompts P; The above mixing matrix weights the regional information, which is a process of multiplying matrix elements, and requires tensor expansion and broadcasting. The obtained mixing matrix is a coarse-grained information fusion; the Language Sematic Attention (LSA) module is a channel-dimensional max-sigmoid attention, which refines the coarse-grained mixing matrix by constraining the distribution of channel directions, thereby alleviating the feature aliasing problem; the coarse-grained mixing matrix is used as a query input, and the regional hint and category embedding are regarded as two feature inputs.
6. A visual language object detector Zone-YOLO based on region cue as claimed in claim 1, It is characterized in that The Zone Head is used for regression tasks, which captures the region-category-entity triple co-occurrence information by fusing the entity information of the image feature I with the region embedding Z; The Zone Head is a detection head based on regional prompts, including a regression branch detection head, a detection head target frame, a self-attention module, an aggregation, and an auxiliary branch. The self-attention module includes "regional dimension self-attention" and "category dimension self-attention"; The regression branch detection head is used to fuse the input image features with the zone embedding to obtain the region-category-entity co-occurrence features, and the co-occurrence information is interacted through the self-attention module to avoid direct matching of text features and image features, while the auxiliary branch is used to maintain resistance to information loss during the learning process.
7. A visual language object detector Zone-YOLO based on region cue as claimed in claim 6, It is characterized in that The regression branch detection head, in the single-stage anchor-free Zone Head, has the classification and regression tasks completely decoupled. Each position of the image feature represents an entity to be regressed, and the channel represents the obtained BBox coordinates. The self-attention module: Due to the introduction of new dimensions, the fusion operation requires a trade-off in computational effort; when performing matrix multiplication on two tensors, the channels are eliminated to obtain the information mixing matrix The region corresponds to dimension K, the category corresponds to dimension N, and the entity corresponds to dimension HW; the two self-attention weighted operations of "region dimension self-attention" and "category dimension self-attention" interact with each other to realize the self-attention between entity and category, entity and region, and guide the model to align the feature space of the information mixing matrix MI; the structure of "region dimension self-attention" and "category dimension self-attention" is the same; unlike using the information mixing matrix MI for the enhancement of a single modality, here the max-sigmoid attention is performed on the information mixing matrix MI in turn; in the region dimension self-attention, the attention weights related to the region and entity are obtained by compressing the category dimension, and in the category dimension self-attention, the attention weights related to the category and entity are obtained by compressing the region information, and these attention weights are loaded on the information mixing matrix to enhance the co-occurrence information; The above process is expressed by formula (5) and formula (6): Where j represents the region dimension, i represents the category dimension, δ represents the sigmoid function, and Conv represents 1×1 convolution; The aggregation: The region-category-entity co-occurrence information obtained through the aggregation operation is entered into the image features to optimize the regression branch of the Zone Head. Specifically, the category dimension is first eliminated using zone embeddings, and then the region information is supplemented into the image features using max-sigmoid attention according to formula (3). The auxiliary branch provides explicit supervision for zone embeddings and uses mean square error (MSE) loss to minimize the distance between the enhanced zone embedding Z' and the zone embedding Z, where the enhanced zone embedding It uses image features to eliminate entity dimensions and obtain enhanced zone embeddings. By aligning the zone embeddings before and after the Zone Head, it alleviates the problem of feature sparsity in the process of mixed information transmission, enhances the stability of features after dimension elimination, and maximizes the regularization effect of the network.
Citation Information
Patent Citations
Text image generation method based on target detection
CN113362416A
Visual language task processing system, training method and device, equipment and medium
CN113792112A
CLIP-based end-to-end scene text recognition method
CN117058667A
Scene graph generation method based on multi-modal contrast learning
CN117710780A
Short-time natural language target tracking method based on visual language large model
CN117746024A
Cited By
Multi-scale visual positioning method and system based on semantic consistency guidance
CN121353998A
BIM (Building Information Modeling) defect detection method based on digital twinning
CN122175874A
