Text guidance-based space-based visual angle image semantic segmentation and target detection collaboration method

Abstract

The invention relates to a space-based visual angle image semantic segmentation and target detection cooperation method based on text guidance, and belongs to the field of image target detection and segmentation. The method comprises the following steps: constructing a data set by using space-based view angle image data and text data, and performing super-pixel division on an image to obtain a segmentation mask; encoding the image data and the text data in the data set to obtain corresponding visual codes and text codes; performing cross-modal coding on the visual codes and the text codes to obtain cross-modal visual features and cross-modal text features; carrying out sampling and aggregation by using the segmentation mask and the cross-modal visual features to obtain super-pixel aggregation features; and updating a query vector and predicting a text guide frame at the target detection branch according to the cross-modal vision and text features, generating a prediction mask by using the cross-modal text features and the super-pixel aggregation features at the image segmentation branch, and aligning the two branches by using the query vector and the prediction mask to output a result. The calculation amount is reduced, and the task efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image target detection and segmentation; in particular, it relates to a text-guided air-based view image semantic segmentation and target detection collaborative method. BACKGROUND

[0002] In the field of image processing, target detection and semantic segmentation are two key tasks aimed at identifying and locating specific targets from images. Target detection determines the location and category of a target by drawing bounding boxes in the image, while image segmentation further precisely divides each pixel of the target from the background, providing a foundation for more in-depth image analysis and understanding, and is widely used in automatic driving, security monitoring, medical image analysis, and other fields. However, air-based view images often carry rich information, with high resolution and rich details, capable of capturing tiny features on the ground, such as early signs of crop disease in crop health monitoring images. At the same time, the shooting scene is complex and varied, possibly covering natural landscapes, urban buildings, farmland, and other environments, with a wide variety of target types, such as pedestrians, vehicles, buildings, or wildlife, and various vegetation. These characteristics make air-based view images have important application value in multiple fields, but also pose challenges to image processing.

[0003] Currently, air-based view image target detection methods have made significant progress, mainly benefiting from the development of deep learning techniques, such as Faster R-CNN, YOLO series, and SSD, which are widely used in air-based view image target detection tasks. However, air-based view images have complex backgrounds and dynamic noise, which can affect the detection accuracy of algorithms and lead to false positives and false negatives. In recent years, deep learning-based semantic segmentation methods have gradually become the mainstream method for air-based view image segmentation tasks, with common models including encoder-decoder architecture, multi-scale and feature fusion strategy, relationship modeling method, visual transformer architecture, and lightweight method. For example, U-Net, DeepLab, and other models perform well in air-based view image segmentation tasks, significantly improving segmentation accuracy through multi-scale feature fusion. However, when applied to air-based view images, their computational resource requirements are high, and the accuracy and completeness of segmentation are affected. Target detection and image segmentation have complementary characteristics: target detection can quickly process a large number of pixels to provide initial positioning information, while image segmentation can provide pixel-level boundary information; target detection usually works at a larger scale, while image segmentation is more suitable for processing more detailed edge information; target detection focuses more on class information, while image segmentation focuses more on target environment. Therefore, the combination of the two can complement each other, effectively improving the detection quality of air-based view images and expanding application scenarios.

[0004] Early target detection and image segmentation collaborative methods mainly use a cascade structure, that is, one task is performed first, and then the results are used to perform another task. For example, the TGNN model first performs a segmentation task, and then uses the results to perform a detection task. The disadvantage of this method is that the performance of the previous task directly affects the performance of the subsequent task, resulting in limited benefits of collaborative training. Some studies attempt to use a single model to handle both tasks simultaneously, such as 3DRefTR, which uses a cascade structure to handle target detection and segmentation tasks, but this design limits the interaction between the two tasks, resulting in poor performance. In addition, the requirements for features of target detection and image segmentation tasks are different, which easily leads to task conflicts. In air-based visual image processing, due to high image resolution, complex background, and diverse targets, the impact of task conflicts is more obvious. For example, target detection may have false positives or false negatives due to complex backgrounds, while the segmentation task may be difficult to process in real time due to the computational requirements of high-resolution images. As two different tasks, the features and limitations of image segmentation and target detection pose challenges to their collaboration. SUMMARY

[0005] In view of the above analysis, the present application aims to disclose an air-based visual image semantic segmentation and target detection collaborative method based on text guidance; the mutual verification of collaborative tasks is realized, which makes up for the respective defects of image segmentation and target detection, and at the same time, the features of the encoder sampling points are enriched by means of superpixel feature aggregation, thereby improving the real-time performance and segmentation performance.

[0006] The air-based visual image semantic segmentation and target detection collaborative method based on text guidance disclosed by the present application comprises:

[0007] Step S1, constructing a data set using air-based visual image data and corresponding text data, and performing superpixel division on all images to obtain a segmentation mask;

[0008] Step S2, encoding the air-based visual image data and text data in the data set to obtain corresponding visual encoding and text encoding;

[0009] Step S3, cross-modal encoding the visual encoding and text encoding, and interactive fusion of visual and text features based on an attention mechanism to obtain cross-modal visual features and cross-modal text features;

[0010] Step S4, superpixel down-sampling and feature aggregation using the segmentation mask and cross-modal visual features to obtain superpixel aggregated features;

[0011] Step S5, in the target detection branch, the cross-modal visual feature and the cross-modal text feature are updated through the Transformer decoding cycle to update the cross-modal feature query vector and predict the text guiding box; in the image segmentation branch, the cross-modal text feature and the superpixel aggregation feature are used to iteratively generate the predicted mask; finally, the query vector and the predicted mask are used to align the output results of the two branches.

[0012] Further, in step S3, comprising:

[0013] Text self-attention encoding; in the text self-attention encoding, the Transformer encoding part structure omitting the feedforward neural network part is adopted for encoding to obtain the language feature;

[0014] Visual self-attention encoding; in the visual self-attention encoding, the updated visual feature is generated by fusing the basic attention weight and the spatial attention weight;

[0015] Cross-attention encoding; in the cross-attention encoding, one of the language feature and the visual feature is selected as the query variable in turn, cross-attention operation is performed, the obtained feature is added to the original feature as the query variable, layer normalization is performed, further processing is performed through the FFN network, and layer normalization is performed again, and finally the cross-modal visual feature is obtained.

[0016] Further, in step S4, the superpixel downsampling process adopts the superpixel segmentation result mask V s and the cross-modal visual feature generated by the cross-modal encoder, K points within a certain radius of the distance of the superpixel s are selected for feature aggregation, and the superpixel aggregation feature V s ’ is obtained.

[0017] Further, the aggregation formula of the aggregation feature v s ’ in the superpixel aggregation feature V s is:

[0018]

[0019]

[0020]

[0021] wherein, o s is the center position of the superpixel s, is the position of the selected kth point, is the relative position encoding of the selected kth point and the superpixel s encoded by the MLP layer; is the visual feature v k of the selected kth point; is the relative position encoding and the visual feature vk and the aggregated feature v of the K selected points after max-pooling of the sum of the relative position encoding and the visual feature s as the aggregated feature v of the superpixel s s ∈V s ’.

[0022] Further, in step S5, the decoder of the image segmentation branch and the prediction head are respectively a Transformer decoder and a prediction head for calculating similarity; the superpixel aggregated feature V s ’ and the cross-modal text feature T′ are sequentially input into the decoder and the prediction head of multiple image segmentation branches to obtain the decoding result and the image segmentation prediction mask M p .

[0023] Further, in the image segmentation branch, each image in the current training batch is traversed for prediction;

[0024] The prediction head uses similarity calculation to update the attention mask according to the following formula:

[0025] M l =S l ×V s ’

[0026]

[0027] and the formula:

[0028] M p =S best ×V s ’

[0029] W p =Sigmoid(score)

[0030] to generate the prediction mask M p and the adaptive weight M p ;

[0031] wherein S best represents the text query vector with the highest score, and score represents the source weight of the superpixel feature:

[0032] score=Linear(ReLU(Linear(norm(S best ))));

[0033] wherein norm(S best ) is the normalization processing of the tensor S best ; Linear(·) is a linear function;

[0034] The decoder comprises two parts of proportional attention SWA and FFN network; wherein the proportional attention SWA calculates the text query vector according to the following cross-attention formula:

[0035]

[0036] Further, in step S5, the decoder of the target detection branch and the prediction head are respectively a Transformer decoder and an EDA target box prediction head; the cross-modal visual feature V' and the cross-modal text feature T' sequentially pass through the decoder and the prediction head of a plurality of target detection branches to obtain the decoding result and the target detection box B p .

[0037] Further, in the target detection branch, the decoder is composed of four parts of encoding the proposal feature, self-attention mechanism of the query vector, cross-attention mechanism between the query vector and the language vector, cross-attention between the query vector and the vision, and FFN; the proposal feature is the feature output after the cross-modal visual feature V' is queried and processed by the target detection prediction head;

[0038] First, the proposal feature box_pro is encoded into a vector Q of the same dimension as the query vector pos The encoding process is as follows:

[0039] Q pos =Conv1d(ReLU(BatchNorm1d(Conv1d(box pro )))

[0040] Q1=Q pos +Q0

[0041] In the formula, Conv1d and BatchNorm1d represent one-dimensional convolution and batch normalization operation respectively; ReLU is an activation function; Q0 is a query vector generated by querying the cross-modal visual feature V';

[0042] Then, Q1 is taken as the variable Q and K, and the original query vector is taken as V to input a spatial self-attention encoder to obtain a query vector Q2;

[0043] Then, the following is executed:

[0044] Q3=Layernorm(Q2+Dropout(MultiHead(Q2+Q pos ,t′,T′)))

[0045] Q4=Layernorm(Q3+Dropout(MultiHead(Q3+Q pos ,V′,V′)))

[0046] Q box = Layernorm(Q4 + Dropout(FFN(Q4)))

[0047] obtaining an output query vector Q box , query is the query vector output by the last layer.

[0048] Further, in step S5, the query vector query obtained by target detection is used to generate a superpixel segmentation result, which is compared with the error calculation to generate a query mask and applied in the error calculation to align the target detection result with the image segmentation result;

[0049] The formula for generating the query mask is:

[0050] M q = query x V s '

[0051]

[0052] wherein, M q i i represents the i-th element of M q , represents the i-th element of V .

[0053] Further, the objective function in the network model training process in step S5 includes a position alignment loss, a target detection loss, an image segmentation loss, and a semantic alignment loss; by minimizing the loss function value, until the training number reaches the initial set threshold or the value of the loss function reaches the preset value, the network model is trained to obtain the network model parameters;

[0054] wherein,

[0055] the position alignment loss L pos is the KL divergence between the predicted visual distribution and the real text distribution;

[0056] the target detection loss L box includes the L1 loss and the generalized intersection over union loss between the predicted bounding box obtained by each layer of the encoder and the center, size of the target bounding box;

[0057] the image segmentation loss L seg includes the focal loss and the dice loss between the predicted mask M p and the query mask and the focal loss and the dice loss after being weighted by the adaptive weight W p , and the focal loss and the dice loss between the predicted mask M pand generating a query mask focal loss and dice loss between them;

[0058] semantic alignment loss L sem , the mean of the similarity between each candidate object visual feature and text feature and the similarity between each text feature and visual feature;

[0059] The loss function L is obtained by weighted sum of the above losses.

[0060] The present application can realize one of the following beneficial effects:

[0061] The text-guided space-based perspective image semantic segmentation and target detection collaborative method disclosed by the present application reduces the multi-task learning calculation amount and improves the task efficiency through the shared feature extraction network. At the same time, the independent branch design avoids the dependence between tasks, enhances the independence and effectiveness of the tasks, and is beneficial to processing high-resolution space-based perspective images. The features are effectively aggregated by superpixels, and the calculation is simplified in superpixel units, improving the real-time performance of image segmentation characters and being more suitable for real-time requirements. Through the generation of a query mask, the two tasks are accurately aligned, improving the generalization ability and overall performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0063] Figure 1 The text-guided space-based perspective image semantic segmentation and target detection collaborative method flow chart in the embodiments of the present application;

[0064] Figure 2 The visual encoder structure schematic diagram in the embodiments of the present application;

[0065] Figure 3 The cross-modal attention encoding module structure schematic diagram in the embodiments of the present application;

[0066] Figure 4 The prediction network schematic diagram of each image of the image segmentation branch in the embodiments of the present application;

[0067] Figure 5 The overall structure schematic diagram of the target detection branch in the embodiments of the present application;

[0068] Figure 6 The network model structure diagram using modular representation in the embodiments of the present application. DETAILED DESCRIPTION

[0069] Preferred embodiments of the present application are described in detail below with reference to the attached drawings, wherein:

[0070] One embodiment of the present application discloses a text-guided air-based view image semantic segmentation and target detection collaborative method, as shown in Figure 1 The method comprises the following steps:

[0071] Step S1, constructing a data set using air-based view image data and corresponding text data, and performing superpixel division on all images to obtain a segmentation mask;

[0072] Step S2, encoding the air-based view image data and text data in the data set respectively to obtain corresponding visual encoding and text encoding;

[0073] Step S3, cross-modal encoding the visual encoding and the text encoding, and interactive fusion of visual and text features based on an attention mechanism to obtain cross-modal visual features and cross-modal text features;

[0074] Step S4, superpixel down-sampling and feature aggregation using the segmentation mask and the cross-modal visual features to obtain superpixel aggregated features;

[0075] Step S5, in the target detection branch, updating a cross-modal feature query vector and predicting a text-guided box through a Transformer decoding cycle using the cross-modal visual features and the cross-modal text features; in the image segmentation branch, iteratively generating a predicted mask using the cross-modal text features and the superpixel aggregated features; and finally aligning the query vector and the predicted mask to output results of the two branches.

[0076] Specifically, in step S1, existing superpixel division methods such as a simple linear iterative clustering (SLIC) algorithm, a normalized cuts (NCuts) algorithm, and a quick shift can be used to obtain the segmentation mask of the image.

[0077] Specifically, in step S2, the image data is visually encoded through pixel down-sampling; the visual encoder used is as shown in Figure 2 The pixel down-sampling process of the visual encoder comprises the following steps:

[0078] 1) hierarchical sampling and feature aggregation;

[0079] The specific steps are to repeat the following steps with the sampling number nsample being 2048, 1024, 512, and 256 respectively to obtain the index, position, and feature of the 4 layers of key points.

[0080] (1) Sample nsample key points from the current point set by the farthest point sampling method, and obtain the index and position thereof;

[0081] (2) For each key point, select K points in the spatial neighborhood thereof, process the positions and colors vis_feat of the K points as the feature new_vis_feat of the key point;

[0082] The processing formula is:

[0083] new_vis_feat=MaxPool(MLP(vis_feat))

[0084] In the formula, MLP represents a multi-layer perception machine, and MaxPool represents maximum pooling.

[0085] 2) Obtain the visual coding feature through feature propagation;

[0086] In turn, the features of the fourth layer and the third layer are back-propagated to the points of the previous layer, distance-weighted interpolation is performed to obtain the features of 512 and 1024 points, and the index, position and features V after the interpolation of the second layer are output.

[0087] Specifically, in step S2, the RoBERTa model is used for text encoding on the text data.

[0088] In step S3, the visual coding and the text coding are subjected to a cross-modal coding process, including:

[0089] 1) Text self-attention coding; in the text self-attention coding, the Transformer coding part structure omitting the feedforward neural network part is used for coding to obtain the language feature;

[0090] 2) Visual self-attention coding; in the visual self-attention coding, the basic attention weight and the spatial attention weight are fused to generate an updated visual feature;

[0091] 3) Cross-attention coding; in the cross-attention coding, one of the language feature and the visual feature is selected as a query variable in turn, cross-attention operation is performed, the obtained feature is added to the original feature as the query variable, layer normalization is performed, the FFN network is further processed, and layer normalization is performed again, and finally the cross-modal visual feature is obtained.

[0092] The cross-modal attention coding module of the cross-modal coding process is as shown in Figure 3

[0093] ​Specifically, in the text self-attention encoding, the text self-attention encoder is a structure of the Transformer encoding part omitting the feed-forward neural network (FFN) part, and the text feature T output by the text encoder is updated to obtain the language feature T1 through the following network:

[0094] T1 = LayerNorm (T + Dropout (MultiHead (T, T, T)))

[0095] Wherein, MultiHead (Q, K, V) represents a multi-head attention operation, Dropout (x) represents a dropout operation on x, and LayerNorm represents a layer normalization operation.

[0096] Specifically, in the visual self-attention encoding, first, the multi-head attention weight of the visual feature is calculated, then the spatial attention weight is calculated by using the maximum pooling result of V and T and the pair position information P, and the spatial attention weight is fused with the basic attention weight, finally the output is calculated through the fused attention weight, and the residual connection and the layer normalization processing are performed to generate the updated visual feature V1.

[0097] Specifically, in the cross-attention encoding, one of the language feature and the visual feature is selected as the query variable in turn, and the cross-attention operation is performed, the obtained feature is added to the original feature as the query variable, the layer normalization is performed, the FFN network is further processed, and the layer normalization is performed again, and finally the cross-modal visual feature and the text feature That is:

[0098] T2 = LayerNorm (T1 + Dropout (MultiHead (T1, V1, V1)))

[0099] V2 = LayerNorm (V1 + Dropout (MultiHead (V1, T1, T1)))

[0100] T' = LayerNorm (T2 + FFN (T2))

[0101] V' = LayerNorm (V2 + FFN (V2))

[0102] Wherein, FFN (x) = Dropout (Linear (Dropout (ReLU (Linear (x))))) ;

[0103] Here, Linear represents a linear transformation layer, and ReLU represents a ReLU activation function.

[0104] Specifically, in step S4, the superpixel down-sampling process adopts the superpixel segmentation result mask V s The K points are selected within a certain radius of the distance superpixel s to aggregate the features, and the superpixel aggregated feature V s ’ is obtained.

[0105] More specifically, the aggregation formula of the aggregated feature v s in the superpixel aggregated feature V s ’ is as follows:

[0106]

[0107]

[0108]

[0109] wherein o s is the center position of the superpixel s, is the position of the selected kth point, is the relative position encoding of the selected kth point and the superpixel s encoded by the MLP layer; is the visual feature v k of the kth point; is the sum of the relative position encoding and the visual feature v k , and the aggregated feature v s obtained by maximum pooling of the sum of the relative position encoding and the visual feature of the selected K points is taken as the aggregated feature V s of the superpixel s. s ’.

[0110] Specifically, in step S5, the decoder and the prediction head of the target detection branch are respectively a Transformer decoder and an EDA target box prediction head; the cross-modal visual feature V′ and the cross-modal text feature T′ pass through the decoder and the prediction head of multiple target detection branches in turn to obtain the decoding result and the target detection box B p .

[0111] Specifically, in step S5, the decoder and the prediction head of the image segmentation branch are respectively a Transformer decoder and a prediction head for calculating similarity; the superpixel aggregated feature V s ’ and the cross-modal text feature T′ pass through the decoder and the prediction head of multiple image segmentation branches in turn to obtain the decoding result and the image segmentation prediction mask M p .

[0112] As Figure 4As shown, in the image segmentation branch, prediction is performed for each image in the current training batch. For each image, the process is as follows: taking the corresponding cross-modal text feature T' as the original text query vector S0, first, S0 and the superpixel aggregation feature V s The input prediction head obtains an initial attention mask A0, and then performs several decoder, prediction head cycles to update the text query vector S l , the attention mask A l , and the source weight of the superpixel feature. The highest-scored text query vector is calculated using the source weight, and finally the prediction head is used on the text query vector to obtain the prediction mask M p , the adaptive weight W p .

[0113] Specifically, the prediction head uses similarity calculation to update the attention mask according to the following formula:

[0114] M l =S l ×V s ’

[0115]

[0116] And use the formula:

[0117] M p =S best ×W s ’

[0118] W p =Sigmoid(score)

[0119] The prediction mask W p and the adaptive weight W p are generated,

[0120] where S best represents the highest-scored text query vector, and score represents the source weight of the superpixel feature:

[0121] score=Linear(ReLU(Linear(norm(S best ))));

[0122] Where norm(S best ) is the normalization processing of tensor S best ; Linear(·) is a linear function.

[0123] The decoder includes a proportion attention SWA and an FFN network. The proportion attention SWA calculates the text query vector according to the following cross-attention formula:

[0124]

[0125] FFN network structure is the same as that in the cross-modal attention encoding.

[0126] In the target detection branch, the decoder is a Transformer decoder, and the prediction head uses an MLP structure to predict the target box center position and size.

[0127] As shown in Figure 5 , first, top-k (k = 256) cross-modal visual features V' are selected to perform query point generation, a one-dimensional convolution layer with a kernel of 1 is used to project the query point to obtain a query vector Q0, and a prediction head is used to obtain a target box proposal feature (i.e., target box position and size). Subsequently, the proposal feature passes through several decoder layers, each of which is composed of a decoder and a prediction head. The decoder decodes the proposal feature and updates the query vector, and the prediction head updates the proposal feature according to the current query vector. The final obtained query vector is recorded as query.

[0128] Further, the specific structure of the decoder in the target detection branch is composed of four parts: encoding of the proposal feature, self-attention mechanism of the query vector, cross-attention mechanism between the query vector and the language vector, and cross-attention between the query vector and the vision, and FFN; the proposal feature is the feature output after the cross-modal visual feature V' is queried and processed by the target detection prediction head;

[0129] First, the proposal feature box_pro is encoded into a vector Q pos with the same dimension as the query vector. The encoding process is:

[0130] Q pos = Conv1d (ReLU (BatchNorm1d (Conv1d (box pro )))

[0131] Q1 = Q pos + Q0

[0132] In the formula, Conv1d and BatchNorm1d represent one-dimensional convolution and batch normalization operations respectively; ReLU is an activation function; Q0 is the query vector obtained by performing query point generation on the cross-modal visual feature V';

[0133] Subsequently, Q1 is taken as the variable Q and K, and the original query vector is taken as V to input a spatial self-attention encoder to obtain a query vector Q2;

[0134] Then, the following is performed:

[0135] Q3 = Layernorm (Q2 + Dropout (MultiHead (Q2 + Q posT', T')

[0136] Q4 = Layernorm(W3 + Dropout(MultiHead(Q3 + Q pos V', V')

[0137] Q box = Layernorm(Q4 + Dropout(FFN(Q4)))

[0138] obtaining an output query vector Q box query is the query vector output by the last layer;

[0139] The FFN network in the decoder in the target detection branch adopts the same FFN network structure as in the cross-modal attention encoding.

[0140] Specifically, since there is a formal difference between the target detection and the image segmentation result, in step S5, the query vector query of the target detection is used to generate the superpixel segmentation result to perform comparison and error calculation to generate the query mask and the error calculation is applied to align the target detection result and the image segmentation result.

[0141] The formula for generating the query mask is:

[0142] M q = query x V s '

[0143]

[0144] wherein, M q i i represents the i-th element of M q , and represents the i-th element of .

[0145] Further, the objective function in the network model training process in step S5 includes a position alignment loss, a target detection loss, an image segmentation loss, and a semantic alignment loss; by minimizing the loss function value, until the training number reaches an initial set threshold or the value of the loss function reaches a preset value, the network model is trained to obtain the network model parameters.

[0146] wherein,

[0147] The position alignment loss L pos is the KL divergence between the predicted visual distribution and the real text distribution.

[0148] The position alignment loss L posThe purpose is to ensure that the visual feature distribution of the language adjustment matches the distribution described in the query text, and the alignment is achieved by predicting the text position similar to the object.

[0149] The target detection loss L box includes the L1 loss and generalized intersection over union loss between the predicted bounding box obtained by each layer of the encoder and the center and size of the target bounding box.

[0150] The image segmentation loss L seg includes the focal loss and dice loss between the predicted mask M p and the generated query mask and the target mask, and the adaptive weight W p weighted focal loss and dice loss, and the focal loss and dice loss between the predicted mask M p and the generated query mask .

[0151] The semantic alignment loss L sem is the average of the similarity between the visual features of each candidate object and the text features and the similarity between each text feature and the visual features.

[0152] The loss function L is obtained by weighting and summing the above losses.

[0153] In this embodiment, a network model structure represented by modules is adopted, as shown in Figure 6 , including a visual encoder, a text encoder, a cross-modal attention encoding module, a superpixel downsampling module, a target detection branch, an image segmentation branch, and an alignment module.

[0154] Wherein,

[0155] The visual encoder is used to encode the space-based visual angle image data in the data set to obtain visual encoding.

[0156] The text encoder is used to encode the text data in the data set to obtain text encoding.

[0157] The cross-modal attention encoding module is used to encode the visual encoding and the text encoding in a cross-modal manner, and the visual and text features are interactively fused based on an attention mechanism to obtain cross-modal visual features and cross-modal text features.

[0158] The superpixel downsampling module is used to perform superpixel downsampling and feature aggregation using the segmentation mask and the cross-modal visual features to obtain superpixel aggregated features.

[0159] The target detection branch is used to update the cross-modal feature query vector through a Transformer decoding cycle and predict a text guided box based on the cross-modal visual features and the cross-modal text features.

[0160] an image segmentation branch, configured to generate a predicted mask iteratively by using cross-modal text features and superpixel aggregation features;

[0161] an alignment module, configured to align the outputs of the two branches by using a query vector and the predicted mask.

[0162] For more detailed technical details of the network model structure, please refer to the contents of the method in the embodiments.

[0163] To sum up, the method for air-based view image semantic segmentation and target detection collaboration based on text guidance disclosed in the embodiments of the present application reduces the amount of multi-task learning calculation and improves the task efficiency by sharing the feature extraction network. At the same time, the independent branch design avoids the dependence between tasks, enhances the independence and effectiveness of the tasks, and is conducive to processing high-resolution air-based view images. The features are effectively aggregated by using superpixels, and the calculation is simplified in superpixel units, improving the real-time performance of image segmentation characters and better adapting to the requirement of real-time performance. The two tasks are accurately aligned by generating a query mask for task alignment, improving the generalization ability and overall performance of the model.

[0164] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A text-guided based space-based view image semantic segmentation and target detection collaborative method, characterized in that, The method comprises the following steps: Step S1, constructing a data set using the space-based visual angle image data and corresponding text data, and performing superpixel division on all images to obtain a segmentation mask; Step S2, encoding the space-based visual angle image data and text data in the data set to obtain corresponding visual encoding and text encoding; Step S3, cross-modal encoding of the visual encoding and the text encoding, interactive fusion of visual and text features based on an attention mechanism, to obtain cross-modal visual features and cross-modal text features; Step S4, superpixel down-sampling and feature aggregation using the segmentation mask and the cross-modal visual features to obtain superpixel aggregated features; Step S5, in the target detection branch, updating the cross-modal feature query vector and predicting the text guide box through the Transformer decoding cycle; in the image segmentation branch, iteratively generating a predicted mask using the cross-modal text features and the superpixel aggregated features; and finally aligning the query vector and the predicted mask to output the results of the two branches.

2. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 1, wherein in step S3, the method comprises: text self-attention encoding; in the text self-attention encoding, a Transformer encoding part structure without a feedforward neural network part is adopted to obtain language features; visual self-attention encoding; in the visual self-attention encoding, updated visual features are generated by fusing the basic attention weight and the spatial attention weight; cross-attention encoding; in the cross-attention encoding, one of the language features and the visual features is selected as a query variable in turn, cross-attention operation is performed, the obtained features are added to the original features as the query variable, layer normalization is performed, the features are further processed through an FFN network, and layer normalization is performed again, and finally cross-modal visual features are obtained.

3. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 2, wherein 4. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 3, wherein In step S4, the superpixel down-sampling process, using the superpixel segmentation result mask V s With the cross-modal visual features generated by the cross-modal encoder, K points are selected within a certain radius of the distance superpixel s to aggregate features, and the superpixel aggregated features V s ’ are obtained.

5. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 2, wherein Superpixel aggregated features V s Aggregated features V in s The aggregation formula for V is: wherein o s is the center position of the superpixel s, is the position of the selected k-th point, is the relative position encoding of the selected k-th point and the superpixel s encoded by the MLP layer; is the visual feature v k of the selected k-th point; is the relative position encoding and the visual feature v k of the selected k-th point, and v s is the aggregated feature v s of the superpixel s after max-pooling the aggregated feature v s ’ of the relative position encoding and the visual feature of the selected K points.

6. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 5, wherein In step S5, the decoder of the image segmentation branch and the prediction head are respectively a Transformer decoder and a prediction head for calculating similarity; and the superpixel aggregation feature V s The cross-modal text feature T' and the image segmentation branch are sequentially input into the decoder and the prediction head of the image segmentation branch to obtain a decoding result and an image segmentation prediction mask M p . in the image segmentation branch, each image in the current training batch is traversed for prediction; the prediction head adopts similarity calculation, and the attention mask is updated according to the following formula: and the formula is used: M l = S l x V s ' the decoder comprises a proportion attention SWA and an FFN network; wherein the proportion attention SWA calculates the text query vector according to the following cross-attention formula: M p = S best x V s ’ W p = Sigmoid(score) Generating a prediction mask M p and an adaptive weight W p ; where S best represents the highest-scored text query vector, and score represents the source weight of the superpixel feature: score = Linear(ReLU(Linear(norm(S best )))) ; wherein norm(S best ) is a normalization process on the tensor S best ; Linear(·) is a linear function; 7. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 4, wherein 8. The text-guided space-based visual angle image semantic segmentation and target detection collaborative method according to claim 7, wherein In step S5, the decoder of the target detection branch and the prediction head are a Transformer decoder and an EDA target box prediction head, respectively; the cross-modal visual feature V' and the cross-modal text feature T' sequentially pass through the decoder and the prediction head of multiple target detection branches to obtain the decoding result and the target detection box B p . ​ In the target detection branch, the decoder is composed of four parts: encoding of the proposal feature, self-attention mechanism of the query vector, cross-attention mechanism between the query vector and the language vector, and cross-attention between the query vector and the vision, and FFN; the proposal feature is the feature output after querying the cross-modal visual feature V' and processing by the target detection prediction head; First, the proposed feature box_pro is encoded into a vector Q of the same dimension as the query vector pos The encoding process is: Q pos = Conv1d(ReLU(BatchNorm1d(Conv1d(box pro )))) Q1 = Q pos +Q0 In the formula, Conv1d and BatchNorm1d respectively represent one-dimensional convolution and batch normalization operations; ReLU is an activation function; Q0 is a query vector generated by querying the cross-modal visual feature V'; Subsequently, a spatial self-attention encoder is input with Q1 as the variable Q, K, and the original query vector as V to obtain a query vector Q2; Then, the following is executed: Q3 = Layernorm(Q2 + Dropout(MultiHead(Q2 + Q pos , T', T')))) Q4 = Layernorm(Q3 + Dropout(MultiHead(Q3 + Q pos , V', V')))) Q box = Layernorm(Q4 + Dropout(FFN(Q4))) The obtained output query vector Q box , query is the query vector output by the last layer.

9. The text-guided air-based perspective image semantic segmentation and target detection collaborative method according to claim 8, characterized in that, In step S5, the query vector query generated by target detection is used to generate a superpixel segmentation result for comparison and error calculation to generate a query mask and applied in error calculation Alignment of target detection result and image segmentation result is realized. Generating query masks The formula is: M q = query x V s ’ where M q i denotes the i-th element of M q denotes the i-th element of M .​ 10. The text-guided air-based perspective image semantic segmentation and target detection collaborative method according to claim 9, characterized in that, The target function of the network model training process in step S5 includes a position alignment loss, a target detection loss, an image segmentation loss, and a semantic alignment loss; by minimizing the loss function value, until the training number reaches the initial threshold or the loss function value reaches the preset value, the network model is trained, and the network model parameters are obtained; Wherein, Position alignment loss L pos is the KL divergence between the predicted visual distribution and the true text distribution. Target detection loss L box includes L1 loss and generalized intersection over union loss between the predicted bounding box obtained by each layer of the encoder and the center and size of the target bounding box. Image segmentation loss L seg , including the focal loss and dice loss between the predicted mask M p and the target mask M , adaptive weight W p weighted focal loss and dice loss, and the focal loss and dice loss between the predicted mask M p and the generated query mask M ; semantic alignment loss L sem is the average of the similarity between the visual features of each candidate object and the text features and the similarity between each text feature and the visual features. The loss function L is obtained by weighted sum of the above losses.

Images