Zone-YOLO, a zone hint-based visual language target detector

Abstract

The application provides a Zone-YOLO based on a region prompt, which comprises a visual encoder, a text encoder, a Scale-Aware VL-PAN Neck, word embedding, zone prompts, an adapter, a Text Contrastive Head and a Zone Head, wherein the application initiates a scale-aware multi-modal fusion module, fully excavates text features, and learns to seamlessly fuse multi-modal representations at different scales; the application provides a new zone prompt learning module, introduces text features into a regression head, and captures triple co-occurrence information of a region, a class and an instance, so that the positioning performance of the model is significantly improved. A large number of experiments show that the Zone-YOLO achieves competitive results, and the superiority of the closed-set detection fine-tuning based on the pre-trained VLOD is proved.

Description

TECHNICAL FIELD

[0001] The present application relates to a Zone-YOLO, which is a visual language target detection implementation scheme based on region prompt. BACKGROUND

[0002] Target detection is an important task of machine vision, which is used for identifying and positioning the region of interest (ROI) in the image, and is also the basis for many tasks such as image understanding and visual positioning. In the past few decades, great breakthroughs have been made in target detection. These methods can be roughly divided into two-stage detectors represented by Faster RCNN, and single-stage detectors represented by DETR series and YOLO series. The former divides the target detection into two sub-tasks of region prompt and prompt optimization, but the inference speed is slow. The latter fuses the detection process into one stage, uses an end-to-end architecture and a lightweight backbone, and performs well in real-time target detection, and is widely used in intelligent transportation, defect detection and remote sensing technology and other fields.

[0003] Vision-Language Models (VLMs) can obtain more general and robust feature representations by fusing multi-modal information. The target detection method based on Vision-Language Models (VLOD) obtains more semantic information from text through a text encoder, alleviates the semantic deficiency problem existing in single visual modal method, and thus significantly improves the detection effect.

[0004] There are two problems worth paying attention to in the research of VLOD:

[0005] Problem 1) The inconsistency between modalities is inherent. Therefore, efficient fusion of visual and language features is the key to whether the model can fully utilize feature information and enhance its representation ability. In view of the first problem, existing methods ignore the distinction between macro and micro concepts of text features, and these concepts are not distinguished from visual features of different scales, which will disturb the image features and cause the deterioration of modality gap.

[0006] Problem 2) Most VLOD methods focus on contrastive learning of text features for classification branches, and do not fully exploit the role of text information for regression branches.

[0007] Conventional VLOD methods ignore the use of language features for target frame regression:

[0008] (i) Describes the problem of reference difficulty, and the method of random matching causes the detector to learn only part of the supervised signal of positive samples each time.

[0009] (ii) The target to be detected cannot be known in advance, so the class-related region prompt cannot be directly used by the present application. SUMMARY

[0010] In order to overcome the shortcomings of the prior art, the present application discloses a visual language target detector based on region prompt Zone-YOLO.

[0011] Technical scheme:

[0012] A visual language target detector based on region prompt Zone-YOLO, comprising a visual encoder, a text encoder, a Scale-Aware VL-PAN Neck, a word embedding, a zone prompt, an Adapter, a Text Contrastive Head and a Zone Head, wherein:

[0013] The visual encoder uses a pre-trained YOLOv8 Backbone, and the output image encoding is a multi-scale image feature;

[0014] The text encoder uses a CLIP text encoder, and the input class name and region phrase are encoded and output as word embedding and zone prompt;

[0015] The Scale-Aware VL-PAN Neck takes the multi-scale image feature output by the visual encoder and the word embedding output by the text encoder as input, and realizes the alignment and fusion of multi-modal and multi-scale features;

[0016] The Text Contrastive Head takes the output of the Scale-Aware VL-PAN Neck and the word embedding as input, calculates the similarity of the image feature and the text feature, and performs contrastive learning;

[0017] The Adapter takes the output word embedding and zone prompt of the text encoder as input, captures the co-occurrence information of the class and the region, and obtains the class-related region embedding Class-specific zone embedding;

[0018] The Zone Head takes the output of the Scale-Aware VL-PAN Neck and the Adapter as input, captures the triple co-occurrence of region-class-entity, and further guides the detector to better regress.

[0019] The application discloses a visual language target detector Zone-YOLO realized by a YOLO structure, and the main innovation points and beneficial effects are as follows.

[0020] 1. The application first creates a scale perception multi-modal fusion module, fully excavates text features, and learns to seamlessly fuse multi-modal representations under different scales.

[0021] 2. The application proposes a new region prompt learning module, introduces text features into a regression head, and captures region-class-instance triple co-occurrence information.

[0022] 3. Fine-tuning is performed on a COCO benchmark dataset, and the superior performance of the Zone-YOLO proposed in the application is proved. Compared with a baseline model, the Zone-YOLO in the application is obviously improved in various indicators, and the AP indicator is improved by 1.7, reaching 55.1. Compared with other similar algorithms, the Zone-YOLO in the application achieves competitive results. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a whole structure schematic diagram of the Zone-YOLO in the application.

[0024] Figure 2 It is a multi-modal attention mechanism schematic diagram. Wherein: (a) is a channel direction double-flow multi-modal attention mechanism of a SAMF sub-module with channel enhancement, and (b) is a channel direction double-flow multi-modal attention mechanism of a SAMF sub-module with mode enhancement. In the figure, I represents image features, and T represents text features, wherein FFN (feedforward neural network) shares parameters for the two modes, and a learnable parameter theta is introduced to control the skip connection.

[0025] Figure 3 It is an Adapter used for obtaining class-related region embedding schematic diagram. Wherein P represents a region prompt, W represents a word embedding, and LSA represents a Language Semantic Attention.

[0026] Figure 4 It is a self-attention module schematic diagram. Wherein: (a) is a self-attention module for MI, and (b) is aggregation of MI into image features. In the figure, I represents image features, and Z represents region embedding. DETAILED DESCRIPTION

[0027] The technical solutions provided by the application will be further described below with reference to specific embodiments and the accompanying drawings. The advantages and features of the application will be more apparent in combination with the following description.

[0028] Model structure

[0029] The whole structure of the Zone-YOLO is as followsFigure 1 as shown:

[0030] A Zone-YOLO based on region prompts, comprising a visual encoder, a text encoder, a Scale-Aware VL-PAN Neck, word embedding, zone prompts, an Adapter, a Text Contrastive Head, and a Zone Head, wherein:

[0031] The visual encoder uses a pre-trained YOLOv8 Backbone, and the output image encoding is a multi-scale image feature.

[0032] The text encoder uses a CLIP text encoder, and the input class name and region phrase are encoded as word embedding and zone prompts.

[0033] The Scale-Aware VL-PAN Neck is used to realize the alignment and fusion of multi-modal multi-scale features, which is one of the key innovations of the present application.

[0034] The Scale-Aware VL-PAN Neck includes a scale-aware modal fusion (SAMF) module, a channel-enhanced SAMF submodule, and a modal-enhanced SAMF submodule.

[0035] The scale-aware modal fusion (SAMF) module further includes a plurality of modal mixing operation SAMF submodules connected in series.

[0036] The channel-enhanced SAMF submodule and the modal-enhanced SAMF submodule each include a corresponding multi-modal attention mechanism.

[0037] Wherein:

[0038] First, the scale-aware modal mixing feature matrix (MI) is obtained through the modal mixing operation SAMF submodule of the scale-aware modal fusion (SAMF) module, the features are supplemented and interacted, and the alignment and fusion of visual and language features are realized.

[0039] Then, based on the multi-modal attention mechanism of the channel-enhanced SAMF submodule and the modal-enhanced SAMF submodule, the image and text features are enhanced using the modal mixing feature matrix (MI).

[0040] The modal mixing operation SAMF submodule:

[0041] Firstly, the image and text features are projected into the same feature space (default both channel numbers have been aligned) to get the coarse-grained "modal mixed feature matrix MI" modal fusion feature matrix where HW represents the spatial dimension of the image, N represents the category dimension, and C represents the channel number.

[0042] Then, the scale-aware query is loaded as a mask on the modal fusion feature matrix MI, and the corresponding scale signal is activated and the signals of other scales are suppressed by the scale-aware query SQ, so that the scale-aware modal mixed feature is obtained. The whole process is described by formula (1):

[0043] MI = Conv(reshape(SQ)·(W x I)) (1)

[0044] where Conv is used to adjust the feature channel number, and the regularization and affine operation are omitted in the formula. The scale-aware query SQ can be a learnable parameter matrix or a spatial attention weight of the image. H represents the height of the image, and W represents the width of the image.

[0045] The scale-aware modal mixed modal fusion feature matrix MI is a coarse fusion feature.

[0046] The channel-enhanced SAMF sub-module and the modal-enhanced SAMF sub-module use two kinds of multi-modal attention mechanisms to combine the scale-aware modal fusion feature matrix MI and perform feature interaction and enhancement in the channel and modal dimensions, respectively.

[0047] The channel-enhanced SAMF sub-module is a channel-directional double-flow multi-modal attention mechanism that gradually refines and fuses the modal fusion feature matrix MI into the text stream and the image stream, thereby realizing adaptive feature enhancement, as shown in Figure 2 (a). More specifically, first, perform a channel attention-like operation on the modal fusion feature matrix MI, use global average pooling (GAP) to compress the spatial and category dimensions, pass through an FFN (feedforward neural network) and a Sigmoid to obtain an attention vector with a length of C, and use the attention vector as a global distribution response in the Channel-wise. Then, the attention vector is multiplied with the image and text features element by element to complete the attention weighting. This is represented by formula (2):

[0048]

[0049] where θ1 and θ2 are used to control the skip connection, I represents the image feature, and T represents the text feature.

[0050] The modal enhanced SAMF sub-module uses the spatial and category dimensions in the modal fusion feature matrix MI to guide the fusion of the two modalities, and scale-aware text is guided to be added to the image features, and scale-aware image is guided to be added to the text features, as shown in Figure 2 (b). Specifically, the maximum value is selected in the category direction of the modal fusion feature matrix MI, and then the sigmoid operation is used to obtain the attention weight for aggregating the text semantic information into the image features. The global average pooling and sigmoid operation are used in the spatial direction of the modal fusion feature matrix MI to aggregate the image feature information and the text features, and the above process is represented by formula (3):

[0051]

[0052] wherein δ represents the sigmoid operation, θ1 and θ2 are used to control the skip connection, I represents the image features, T represents the text features, N represents the category dimension, and HW represents the spatial dimension of the image.

[0053] The Text Contrastive Head (text-contrastive head) is a module structure in the existing method "YOLO-World: Real-Time Open-Vocabulary Object Detection" [1] , which takes the output of the Scale-Aware VL-PAN Neck and the word embeddings as inputs, calculates the similarity of the image features and the text features, and performs contrastive learning.

[0054] The relationship between each component is as follows:

[0055] The output of the zone prompts is connected to the adapter, and the output of the adapter is connected to the zone head, which is used to capture the triple co-occurrence of the region-class-entity. The zone prompts explicitly introduce the zone information into the zone head to guide the regression of the target box. The class-independent zone prompts are used to solve the problem of reference difficulty, the adapter obtains the class-dependent zone embeddings to capture the class-region co-occurrence, the zone head obtains the region-class-entity co-occurrence features, and finally, the self-supervised auxiliary branch is used to enhance the stability of the zone embeddings.

[0056] The Adapter is used to capture the class-zone co-occurrence information of word embeddings and zone prompts. The Adapter captures the co-occurrence information of classes and zones to obtain "class-specific zone embeddings".

[0057] The Zone Head fuses the zone embeddings output by the Adapter with the output of the Scale-Aware VL-PAN Neck to guide the detector to better regress.

[0058] The Adapter, i.e., the adapter, is composed of a single-layer Transformer module and a Language Sematic Attention (LSA) module. As shown in Figure 3

[0059] zone prompts, including a class-agnostic zone prompts submodule, a class-specific zone embedding submodule;

[0060] The class-agnostic zone prompts submodule divides the image into K regions, each region corresponding to a fixed directional noun as a description, and sends the K descriptions into the text encoder to obtain class-agnostic zone prompts where C is the number of channels, consistent with T. As an example, by way of illustration but not limitation, K takes the value of 9, and the 9 regions are upper left, top center, upper right, middle left, center, middle right, lower left, bottom center, and lower right. Because the zone prompts obtained using text descriptions contain more semantic information, numbering is not used as a region description as in other methods.

[0061] The class-agnostic zone prompts submodule is used to mine the context patterns of the class appearing in the image, thereby helping object localization. In order to make full use of the semantic representation capability of the text encoder and capture the co-occurrence information of classes and zones, the working mechanism of the class-agnostic zone prompts submodule in cooperation with the Adapter is as follows:

[0062] ​Firstly, the category embedding is input into the Transformer module of the Adapter as the value and key matrix, and the region prompt is input as the query matrix, so as to capture the long-range semantic relationship between the category and the region; then, the Language Sematic Attention (LSA) module is used to calculate the mixed matrix of the region prompt and the category embedding information, and the mixed matrix is weighted with the region information to obtain the region embedding related to the category The whole process is represented by formula (4):

[0063]

[0064] Wherein, δ represents the sigmoid operation; MHSA refers to the Multi-Head Self-Attention operation; K is the number of region prompts P, and the value of an embodiment is 9.

[0065] The weighting of the mixed matrix with the region information is a process of multiplying the matrix elements, which needs to be tensor expanded and broadcasted, and the obtained mixed matrix is a coarse-grained information fusion. The Language Sematic Attention (LSA) module is a max-sigmoid attention in the channel dimension, which refines the coarse-grained mixed matrix by constraining the distribution of the channel direction, and then relieves the feature aliasing problem. The coarse-grained mixed matrix is input as the query, and the region prompt and the category embedding are regarded as two feature inputs, and the structure of the Language Sematic Attention (LSA) module and the scale-aware modal fusion (SAMF) module is unified.

[0066] The Zone Head is used for a regression task, and the structure is shown in Figure 1 By fusing the entity information of the image feature I and the region embedding Z, the Zone Head is used to capture the region-category-entity triple co-occurrence information.

[0067] The Zone Head is a region prompt-based detection head, which includes a regression branch detection head, a detection head target frame, a self-attention module, an aggregate, and an auxiliary branch. The self-attention module includes a region dimension self-attention and a category dimension self-attention.

[0068] The input image feature and the zone embedding are fused by the regression branch detection head, so as to obtain the region-category-entity co-occurrence feature, and the co-occurrence information is interacted by the self-attention module, so as to avoid the direct matching of the text feature and the image feature. At the same time, the auxiliary branch is used to maintain the resistance to information loss in the learning process.

[0069] The regression branch detection head is in a single-stage anchor-free Zone Head, in which the classification and regression tasks are completely decoupled, each position of the image feature represents an entity to be regressed, and the channel (usually 4) represents the obtained BBox coordinates.

[0070] The self-attention module includes:

[0071] Due to the introduction of a new dimension, the fusion operation needs to make a trade off in the amount of calculation. When the two tensors are multiplied by a matrix, the information mixing matrix MI is obtained by eliminating the channel where K is the region corresponding dimension (in the embodiment, the value of K is 9), N is the category corresponding dimension, and HW is the entity corresponding dimension. The loss of channel information may exacerbate the gap in the feature space, therefore, the application designs two self-attention weighting operations of “region dimension self-attention” and “category dimension self-attention” to interact information, realizes self-attention of entities and categories, self-attention of entities and regions, and guides the model to align the feature space of the information mixing matrix MI. The structure of “region dimension self-attention” and “category dimension self-attention” is the same. Figure 4 (a) shows the details of the above two operations, unlike the previous use of the information mixing matrix MI for single-modal enhancement, here the max-sigmoid attention is performed on the information mixing matrix MI in turn. In the region dimension self-attention, the attention weight related to the region and the entity is obtained by compressing the category dimension, in the category dimension self-attention, the attention weight related to the category and the entity is obtained by compressing the region information, and the attention weight is loaded on the information mixing matrix to realize the enhancement of co-occurrence information. The above process is represented by formula (5) and formula (6):

[0072]

[0073]

[0074] where j represents the region dimension, i represents the category dimension, δ represents the sigmoid function, and Conv represents the 1*1 convolution.

[0075] The aggregate operation includes:

[0076] The region-category-entity co-occurrence information obtained by the aggregate operation enters the image feature, and the regression branch of the Zone Head (detection head) is optimized, as shown in Figure 4 (b). Specifically, the zone embeddings are used to eliminate the category dimension, and then the max-sigmoid attention is used to supplement the region information to the image feature according to formula (3).

[0077] The auxiliary branch provides explicit supervision for zone embeddings. Mean squared error (MSE) loss is used to minimize the distance between augmented zone embeddings Z’ and zone embeddings Z, where the augmented zone embeddings are obtained by eliminating entity dimension with image features. By aligning zone embeddings before and after Zone Head, the problem of feature sparsity in mixed information passing is alleviated, and the stability of features after eliminating dimension is enhanced, while the network maximizes the effect of regularization.

[0078] Experiments

[0079] Main Results

[0080] Quantitative analysis of Zone-YOLO and comparative methods on COCO and LVIS. For fairness, the results of all comparative methods come from published papers or their released codes.

[0081] Comparative results on COCO dataset.

[0082] Table 1 is a comparison of different YOLO object detectors on COCO. The Zone-YOLO is fine-tuned on COCO train2017 and evaluated on COCO val2017. For fairness, all models use the mask-refine configuration. indicates that the Zone-YOLO is based on “YOLO-World: Real-Time Open-Vocabulary Object Detection” [1] Fine-tune the weights pre-trained on more data. The best results are highlighted explicitly.

[0083]

[0084]

[0085] Table 1 shows the performance of Zone-YOLO and the latest YOLO series methods [1][2][3][4][5] on COCO benchmark. Zone-YOLO and YOLO-World are fine-tuned on pre-trained weights for 80 epochs, while other models are trained from scratch, which requires hundreds of epochs. As can be seen in the table, Zone-YOLO-L achieves comprehensive improvement over the baseline model, especially in AP 75 and AP M, respectively, compared to 2.3 and 1.9 for “YOLO-World: Real-Time Open-Vocabulary Object Detection” [1] In comparison, the present application achieves comprehensive outperformance in all metrics. Thanks to the proposed method and large-scale pre-training, Zone-YOLO-L is significantly higher than YOLOv9 and YOLOv10 in AP, 75 and AP M metrics by 1.9 and 2.4, 2.8 and 3.4, and 1.9 and 2.6, respectively. As expected, Zone-YOLO- achieves the best results in AP, AP 50 , AP M and AP L . Small and medium-sized models also achieve satisfactory results, although there is still a certain gap in a few metrics. Overall, Zone-YOLO achieves competitive results compared to other models in the YOLO series.

[0086] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the above disclosed technical content should be regarded as an equivalent effective embodiment, and belongs to the protection scope of the technical scheme of the present application.

[0087] [1] Cheng T, Song L, Ge Y, et al. YOLO-World: Real-Time Open-Vocabulary Object Detection [J]. arXiv preprint arXiv:2401.17270, 2024.

[0088] [2] Li C, Li L, Jiang H, et al. YOLOv6: A single-stage object detection framework for industrial applications [J]. arXiv preprint arXiv:2209.02976, 2022.

[0089] [3] G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO,” Jan. 2023. [Online]. Available: https: / / github.com / ultralytics / yolo https: / / github.com / ultralytics / ultralytics.

[0090] [4] Wang C Y, Yeh I H, Liao H Y M. YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information [J]. arXiv preprint arXiv:2402.13616, 2024.

[0091] [5] Wang A, Chen H, Liu L, et al. YOLOv10: Real-Time End-to-End Object Detection [J]. arXiv preprint arXiv:2405.14458, 2024.

Claims

1. A Zone-YOLO based on regional hint visual language target detector, characterized in that, The visual encoder, the text encoder, the Scale-Aware VL-PAN Neck, the word embedding, the zone prompts, the Adapter, the Text Contrastive Head, and the Zone Head, wherein: The visual encoder uses a pre-trained YOLOv8Backbone, and outputs image encoding as multi-scale image features; The text encoder uses a CLIP text encoder, and inputs class names and zone phrases, and encodes outputs as word embedding and zone prompts; The Scale-Aware VL-PAN Neck takes the multi-scale image features output by the visual encoder and the word embedding output by the text encoder as inputs, and realizes alignment and fusion of multi-modal and multi-scale features; The Text Contrastive Head takes the output of the Scale-Aware VL-PAN Neck and the word embedding as inputs, calculates the similarity of image features and text features, and performs contrastive learning; The Adapter takes the word embedding and zone prompts output by the text encoder as inputs, captures the co-occurrence information of categories and zones, and obtains class-specific zone embedding; The Zone Head takes the outputs of the Scale-Aware VL-PAN Neck and the Adapter as inputs, captures the triple co-occurrence of zone-category-entity, and guides the detector to better regress.

2. The Zone-YOLO of claim 1, wherein: The Scale-Aware VL-PAN Neck includes a scale-aware modal fusion (SAMF) module, a channel-enhanced SAMF submodule, and a modal-enhanced SAMF submodule; the scale-aware modal fusion (SAMF) module further includes a plurality of modal mixing operation SAMF submodules connected in series; the channel-enhanced SAMF submodule and the modal-enhanced SAMF submodule each include a corresponding multi-modal attention mechanism; The working process includes: First, the scale-aware modal mixing feature matrix (MI) is obtained through the modal mixing operation SAMF submodule of the scale-aware modal fusion (SAMF) module, the features are supplemented and interacted, and the alignment and fusion of visual and language features are realized; Then, based on the multi-modal attention mechanism of the channel-enhanced SAMF submodule and the modal-enhanced SAMF submodule, the image and text features are enhanced using the modal mixing feature matrix (MI). 3.The Zone-YOLO based on zone prompt of claim 2, wherein, the SAMF sub-module of the modal mixed operation: First, the image and text features are first projected into the same feature space to obtain a coarse-grained "modal mixed feature matrix MI" modal fusion feature matrix where HW represents the spatial dimension of the image, N represents the category dimension, and C represents the number of channels. Subsequently, the scale-aware query As a mask is loaded onto the modal fusion feature matrix MI, the corresponding scale signal is activated and the signals of other scales are suppressed by the scale-aware query SQ, so as to obtain the scale-aware modal mixed feature; the whole process is described by formula (1): MI=Conv(reshape(SQ)·(W×I))(1)wherein, Conv is used to adjust the number of feature channels; H represents the height of the image; and W represents the width of the image. 4.The Zone-YOLO based on zone prompt of claim 2, wherein, the SAMF sub-module of the channel enhancement and the SAMF sub-module of the modal enhancement use two kinds of multi-modal attention mechanisms to combine the scale-aware modal fusion feature matrix MI and respectively perform feature interaction and enhancement in the channel and modal dimensions, wherein: the SAMF sub-module of the channel enhancement is a double-flow multi-modal attention mechanism in the channel direction, which gradually refines and fuses the modal fusion feature matrix MI into the text stream and the image stream, so as to realize adaptive feature enhancement, and specifically: first, the modal fusion feature matrix MI is subjected to an operation similar to channel attention, the spatial and class dimensions are compressed using global average pooling (GAP), and the length C attention vector is obtained through FFN (feedforward neural network) and Sigmoid, which is used as a global distribution response in the Channel-wise; then the attention vector is respectively multiplied with the image and text features to complete attention weighting; which is expressed by equation (2): wherein, θ1 and θ2 are used to control the skip connection, I represents the image feature, and T represents the text feature; the SAMF sub-module of the modal enhancement uses the spatial and class dimensions in the modal fusion feature matrix MI to guide the fusion of the opposite modal, and the scale-aware text guide is added to the image feature, and the scale-aware image guide is added to the text feature, and specifically: the maximum value is selected in the class direction of the modal fusion feature matrix MI, and then the sigmoid operation is used to obtain the attention weight, which is used to aggregate the text semantic information into the image feature; the image feature information and the text feature are aggregated using the global average pooling and the sigmoid operation in the spatial direction of the modal fusion feature matrix MI, and the above process is expressed by equation (3): wherein, δ represents the sigmoid operation, θ1 and θ2 are used to control the skip connection, I represents the image feature, T represents the text feature, N represents the class dimension, and HW represents the spatial dimension of the image. 5.The Zone-YOLO based on zone prompt of claim 1, wherein, the Adapter is composed of a single-layer Transformer module and a Language Sematic Attention (LSA) module; the zone prompts include a class-agnostic zone prompts sub-module and a class-dependent zone embedding sub-module. The class-agnostic zone prompts module divides the image into K regions, each region corresponds to a fixed directional noun as a description, and K descriptions are input into the text encoder to obtain class-agnostic zone prompts where C is the number of channels, consistent with T; The category-independent region prompt sub-module is used to mine the context mode of the category appearing in the image, thereby helping to locate the object; in order to make full use of the semantic representation capability of the text encoder and capture the co-occurrence information of the category and the region, the working mechanism of the category-independent region prompt sub-module cooperating with the Adapter is as follows: Firstly, the category embedding is input into the Transformer module of the Adapter as the value and key matrix, and the region prompt is input as the query matrix, to capture the long-range semantic relationship between the category and the region; then, the Language Sematic Attention (LSA) module is used to calculate the mixed matrix of the region prompt and the category embedding information, and the mixed matrix is weighted with the region information to obtain the region embedding related to the category The overall process is represented by equation (4): Wherein, δ represents a sigmoid operation; MHSA represents a multi-head self-attention mechanism (Multi-Head Self-Attention) operation; K is the number of region prompts P; The above mixing matrix performs weighting on the region information, and the process of multiplying the matrix elements needs to be tensor expanded and broadcasted, so that the obtained mixing matrix is a coarse-grained information fusion; the Language Sematic Attention (LSA) module is a channel-dimension max-sigmoid attention, which refines the coarse-grained mixing matrix by constraining the distribution of the channel direction, thereby relieving the feature aliasing problem; the coarse-grained mixing matrix is taken as a query input, and the region prompt and the category embedding are taken as two feature inputs.

6. The Zone-YOLO based on a region prompt, according to claim 1, wherein, The Zone Head is used for a regression task, and the entity information of the image feature I is fused with the region embedding Z to capture the region-category-entity triple co-occurrence information; The Zone Head is a region prompt-based detection head, including a regression branch detection head, a detection head target frame, a self-attention module, an aggregate, and an auxiliary branch; the self-attention module includes a region-dimension self-attention and a category-dimension self-attention; The regression branch detection head is used to fuse the input image feature with the zone embedding, so as to obtain region-category-entity co-occurrence features, and the self-attention module is used to interact the co-occurrence information, thereby avoiding the direct matching of the text features and the image features, and the auxiliary branch is used to maintain the resistance to information loss in the learning process.

7. The Zone-YOLO based on a region prompt, according to claim 6, wherein, The regression branch detection head is used in a single-stage anchor-free frame Zone Head, and the classification and regression tasks are completely decoupled; each position of the image feature represents an entity to be regressed, and the channel represents the obtained BBox coordinates; The self-attention module includes: Due to the introduction of new dimensions, the fusion operation needs to make a compromise in the amount of calculation; when performing matrix multiplication on two tensors, the channel is eliminated to obtain an information mixing matrix where the region corresponds to the dimension K, the category corresponds to the dimension N, and the entity corresponds to the dimension HW; the two self-attention weighting operations of "region dimension self-attention" and "category dimension self-attention" perform information interaction, realize self-attention of entities and categories, self-attention of entities and regions, and guide the model to align the feature space of the information mixing matrix MI; "region dimension self-attention" and "category dimension self-attention" have the same structure; unlike using the information mixing matrix MI to enhance a single modality, max-sigmoid attention is performed on the information mixing matrix MI in turn here; in the region dimension self-attention, the category dimension is compressed to obtain attention weights related to the region and the entity, and in the category dimension self-attention, the region information is compressed to obtain attention weights related to the category and the entity; loading these attention weights on the information mixing matrix realizes the enhancement of co-occurrence information; The above process is represented by formula (5) and formula (6): Wherein, j represents the region dimension, i represents the category dimension, δ represents the sigmoid function, and Conv represents a 1×1 convolution; The aggregate: The region-category-entity co-occurrence information obtained through the aggregation operation enters the image feature, and the regression branch of the Zone Head is optimized, specifically: first, the zone embeddings are used to eliminate the category dimension, and then, according to formula (3), the region information is supplemented into the image feature using max-sigmoid attention; The auxiliary branch provides explicit supervision for zone embeddings; uses a mean squared error (MSE) loss to minimize the distance between the augmented zone embedding Z' and the zone embedding Z, where the augmented zone embedding is obtained by eliminating the entity dimension using image features; by aligning the zone embeddings before and after the Zone Head, the problem of feature sparsity in the mixed information transmission process is alleviated, the stability of the features after eliminating the dimension is enhanced, and the regularization effect of the network is maximized.

Images