Image feature extraction and analysis method based on multi-region gating fusion

Abstract

The application relates to the technical field of computer vision, in particular to an image feature extraction and analysis method based on multi-region gate fusion, which comprises the following steps: obtaining a target region mask of an image, generating a global image and a plurality of local region images through partition operation, extracting global and local tile features in parallel, and enhancing the context representation of the global tile features in the spatial and channel dimensions; adopting a region gate fusion mechanism to dynamically adjust and fuse the global enhanced features and the local tile features; aggregating the fused features, inputting the aggregated features into an encoder and a structured decoder in combination with a guide query tensor generated by a memory guide module for interactive processing; outputting a multi-dimensional structured information prediction through a multi-task prediction head, and outputting a structured analysis result; the application solves the problems of insufficient attention to local fine-grained targets in image fine-grained analysis, imbalance between global and local feature fusion, and lack of structured semantic description of output results in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image feature extraction and analysis method based on multi-region gating fusion. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies and the explosive growth of image data, traditional analysis methods based on manual screening (such as medical image reading and industrial quality inspection) are no longer able to meet the growing practical application demands. Existing image analysis technologies mainly rely on deep convolutional neural networks or visual Transformers to extract global features from the entire image, lacking effective utilization of prior knowledge of the target region's structure, resulting in insufficient accuracy in identifying fine-grained targets or anomalies in specific regions.

[0003] Refined image analysis aims to uncover complementary information between global context and local detail features, using deep learning techniques to achieve precise localization of key targets and multi-dimensional attribute assessment. Traditional feature extraction methods typically employ simple global average pooling or direct feature concatenation strategies, failing to dynamically balance the weights of global overview and local details during feature fusion. This can easily lead to background noise interference or loss of crucial fine-grained features. Furthermore, existing methods often limit their output to a single classification label, struggling to generate structured information encompassing multi-dimensional attributes such as target location, attribute severity, and state category (e.g., negative / positive, qualified / defective). This fails to meet the stringent requirements of interpretability and comprehensiveness in high-precision image analysis. Therefore, current image refinement analysis techniques suffer from insufficient attention to local fine-grained targets, an imbalance in global and local feature fusion, and a lack of structured semantic description in the output results. Summary of the Invention

[0004] The purpose of this invention is to provide an image feature extraction and analysis method based on multi-region gated fusion, which aims to solve the problems in the existing technology of insufficient attention to local fine-grained targets in image fine analysis, imbalance between global and local feature fusion, and lack of structured semantic description in the output results.

[0005] To achieve the above objectives, this invention provides an image feature extraction and analysis method based on multi-region gated fusion, comprising: The image to be analyzed is used to generate a target region mask using a pre-trained region extraction model. The mask is then decomposed into a global image and multiple local target sub-images to construct hierarchical input data containing prior knowledge of the target region structure. The global image and multiple local target sub-images are input into a visual feature extractor with shared weights to perform depth feature extraction, generating global patch features and local patch features corresponding to each region respectively. Global tile features are input into a dual attention module and processed in parallel by location and channel attention branches to capture long-range spatial dependencies and channel semantic associations, resulting in global enhanced features. A regional gating fusion mechanism is adopted, in which the regional gating network calculates the dynamic weight coefficients of global enhanced features and local features, and the two are weighted and fused to generate regional fusion features that take into account both global and local features; Multi-region fusion features are input into the hybrid expert style aggregation module, which learns and weights them to generate unified fusion visual features, which are then fed into the Transformer encoder for encoding to obtain high-order semantic feature representations; The guided query tensor generated by the memory guidance module through the unified fusion of visual features is used as the query input of the cross-attention module in the structured decoder. It interacts non-autoregressively with the semantic features output by the encoder to output the target query features. The target query features output by the structured decoder are fed into multiple parallel prediction heads. Each prediction head shares the same input features but performs inference independently, thereby completing multi-dimensional structured information prediction and outputting structured analysis results.

[0006] The specific steps involved in generating a target region mask from the image to be analyzed using a pre-trained region extraction model, decomposing the image into a global image and multiple local target sub-images through mask operations, and constructing hierarchical input data containing prior knowledge of the target region structure include: The image to be analyzed undergoes preprocessing operations including size normalization to 224×224 pixels, pixel value normalization, and data augmentation. A pre-trained lightweight region extraction model is used to generate a target region mask through an encoder-decoder structure. Based on the mask and the original image, a Hadamard product operation is performed to decompose the image into a global image and multiple local sub-images, thereby constructing hierarchical input data containing prior information about the target region structure for subsequent feature extraction.

[0007] The specific steps involved in inputting the global image and multiple local target sub-images into a weighted visual feature extractor for depth feature extraction, generating global patch features and corresponding local patch features for each region, include: The global image and multiple local target sub-images are input into a visual feature extractor with shared weights to perform deep feature extraction. The visual feature extractor uses the convolutional layer of the pre-trained model as the feature encoding backbone network, removes the fully connected classification layer of the original model and adds an adaptive average pooling layer to aggregate the feature maps into a fixed-dimensional feature vector. By sharing network parameters, global patch features containing the overall anatomical structure are extracted from the global image, and local patch features focusing on specific target regions are extracted from the local sub-images.

[0008] The specific steps involved in inputting global tile features into a dual attention module, processing them in parallel through location and channel attention branches, capturing long-range spatial dependencies and channel semantic associations, and obtaining globally enhanced features include: Global tile features are input into the dual attention module. The positional attention branch calculates the product of the input feature matrix and its transpose, performs normalization by scaling factor, and applies the softmax function to generate a spatial attention map. The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix. The two attention branches are integrated through a weighted fusion strategy.

[0009] The specific steps involved in employing a regional gating fusion mechanism, where a regional gating network calculates the dynamic weight coefficients of global and local enhanced features, and then weights and fuses them to generate regional fusion features that take into account both global and local aspects, include: The regional gating fusion mechanism comprises multiple independent channel-level gating fusion modules, each corresponding to a local target region; for the k-th local region, global enhancement features are applied. With the kth local feature Average pooling is performed on the tile dimension M to generate global summary features. Local summary features Where B is the batch size and D is the feature dimension; the two feature sums are added together and then input into a multilayer perceptron (MLP) structure to generate channel-gated weights. The MLP uses a two-layer fully connected network and the gating weights are activated by a sigmoid activation function. The value range is limited to between 0 and 1; channel gating weight After broadcasting to the tile dimension via the dimension expansion operation, we get... Through the soft fusion formula: ; The global enhanced features and the k-th local features are fused through a channel-wise weighted fusion process to generate a region fusion feature that takes into account both global semantic context and local anatomical structure. In the formula, This represents the Hada code product.

[0010] The specific steps involved in the multi-region fusion feature input hybrid expert style aggregation module, which generates unified fused visual features through learning and weighting, and then feeds them into the Transformer encoder for encoding to obtain high-order semantic feature representations include: Multiple sets of region fusion features are input into a hybrid expert-style aggregation module for multi-region semantic integration. This module treats each region fusion feature as an independent expert and adaptively calculates the contribution weight of each region to the final diagnosis through a learnable routing network. Average pooling is performed at the tile dimension to obtain region vector representations. Each region vector is then input into an independent linear projection layer to map it into a route score scalar. Route weights are generated using the Softmax normalization function. Extending to the tile dimension yields Using the weighted summation formula: ; Adaptive aggregation of regional fusion features to generate unified fused visual features , in the formula, This represents element-wise multiplication. The unified fused visual features are mapped to a unified hidden dimension space through a linear projection layer and then input into a multi-layer Transformer encoder for deep semantic encoding. Each Transformer encoder layer contains a multi-head self-attention module, residual connections, layer normalization, and a feedforward neural network to generate encoder semantic features containing high-order global semantic information.

[0011] The specific steps for outputting the target query features include: First, the guided query tensor generated by the memory-guided module, which integrates unified visual features, is used as the query input to the cross-attention module in the structured decoder. This tensor interacts non-autoregressively with the semantic features output by the encoder. The unified and fused visual features are fed into the memory guidance module. After obtaining the global visual representation through average pooling, they are input into a multilayer perceptron for feature encoding. The vector of modulation parameters is generated by mapping through a fully connected layer and then split into scaling parameters and bias parameters. The query tensor is conditionally modulated to obtain the guidance query tensor. In the structured decoding stage, the guidance query tensor serves as the query for the first layer of the structured decoder with multi-head cross-attention. The semantic features output by the encoder serve as the key and value. The structured decoder consists of multiple stacked decoding layers. Each layer contains multi-head cross-attention, layer normalization, feedforward network, and Dropout module. The multi-head self-attention structure in the traditional Transformer decoder is removed. A parallel non-autoregressive decoding method is used to model the correspondence between the target query and the image semantic features and output the target query features.

[0012] The specific steps involved in feeding the target query features output by the structured decoder into multiple parallel prediction heads, where each prediction head shares the same input features but performs inference independently, simultaneously completing multi-dimensional structured information prediction and outputting structured analysis results, include: The structured decoder outputs target query features and feeds them into a multi-task prediction head. Parallel sub-networks then perform multi-dimensional structured information prediction, outputting structured analysis results in parallel. The system employs a multi-task joint loss function for end-to-end training, including target category identification, existence judgment, target confidence estimation, attribute state classification, and target location localization. Finally, it outputs structured analysis results in parallel. The joint loss calculation formula is as follows: ; Where N is the total number of prediction heads. Let be the loss function for the i-th task. Let be the loss weight coefficient for the i-th task. The optimal balance is determined through hyperparameter tuning, and the specific form of the loss function for each task is flexibly designed according to the task characteristics. In the subsequent result decoding stage, the system provides an optional conditional filtering mechanism. This mechanism supports flexibly configuring any combination of tasks as preconditions based on task characteristics. When the conditional filtering mechanism is enabled, a binary mask M(k) is introduced for each target k in the decoding stage, defined as: ; in This represents the set of prerequisite tasks. This indicates an indicator function that takes the value 1 when the condition is met and 0 otherwise. The preset threshold is the threshold value for the j-th preceding task.

[0013] This invention discloses an image feature extraction and analysis method based on multi-region gating fusion, comprising: generating a target region mask from the image to be analyzed using a pre-trained region extraction model; decomposing the mask into a global image and multiple local sub-images using a Hadamard operation; constructing hierarchical input data containing prior knowledge of the target region structure; inputting the global and local sub-images into a CNN or ViT visual feature extractor with shared weights for deep feature extraction, generating global patch features and local patch features corresponding to each region; inputting the global patch labels into a dual attention module, which processes them in parallel through position and channel attention branches to capture long-range spatial dependencies and channel semantic associations, thereby obtaining global augmented features. Strong feature representation is employed; a region-gated fusion mechanism is used, where a gating network calculates dynamic weight coefficients for global enhanced features and local patch features, and then weights and fuses them to generate a region-fused feature that balances global and local characteristics. The unified fused visual features, through a memory-guided module, generate a guided query tensor as the query input to the cross-attention module in the structured decoder. This tensor interacts non-autoregressively with the semantic features output by the encoder to output the target query feature. The target query feature output by the structured decoder is then fed into multiple parallel prediction heads. Each prediction head shares the same input features but performs inference independently, simultaneously predicting multi-dimensional structured information and outputting structured analysis results. Based on this method, in the dual attention module, the parallel processing of positional attention and channel attention branches enables the simultaneous capture of long-range spatial dependencies and channel semantic associations of the global image, effectively enhancing the expressive power of global features. In the region-gated fusion mechanism, the gating network adaptively adjusts the weights of global enhanced features and local detail features, overcoming the problem of easy loss of local fine-grained features or separation from the global context in traditional methods, achieving organic complementarity of multi-scale features. In the memory-guided and decoding stages, a guiding query tensor generated by the memory-guided module is introduced as a query for the cross-attention module in the structured decoder, providing accurate prior information guidance for the decoding stage. Combined with a non-autoregressive decoding mode, it breaks the dependency limitations of traditional serial generation and, together with a multi-task prediction head, can output multi-dimensional structured analysis results in parallel. This technical solution fully utilizes prior knowledge of the target region structure in images, significantly improving the feature extraction accuracy and inference analysis efficiency of fine-grained targets (such as medical lesions and industrial defects) in various complex images, providing high-accuracy and high-real-time technical support for the automated analysis of complex visual scenes. This invention solves the problems of insufficient attention to local fine-grained targets, imbalance between global and local feature fusion, and lack of structured semantic description in the output results of existing image refinement analysis. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0015] Figure 1 This is a flowchart of the image feature extraction and analysis method based on multi-region gating fusion proposed in this application.

[0016] Figure 2 This is a network structure diagram of the image feature extraction and analysis method based on multi-region gating fusion proposed in this application.

[0017] Figure 3 This is a schematic diagram of the dual attention enhancement module of the image feature extraction and analysis method based on multi-region gating fusion proposed in this application.

[0018] Figure 4 This is a schematic diagram of the region-gated network for image feature extraction and analysis based on multi-region gating fusion, as described in this application.

[0019] Figure 5 This is a schematic diagram of the hybrid expert style aggregation module of the image feature extraction and analysis method based on multi-region gating fusion in this application.

[0020] Figure 6 This is a schematic diagram of the memory guidance module of the image feature extraction and analysis method based on multi-region gating fusion proposed in this application.

[0021] Figure 7 This is a schematic diagram of the structured decoder based on the image feature extraction and analysis method of multi-region gating fusion proposed in this application.

[0022] Figure 8 This is a schematic diagram of the structure of the multi-task prediction head based on the image feature extraction and analysis method of multi-region gating fusion in this application. Detailed Implementation

[0023] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0024] Please see Figures 1-8 This invention provides an image feature extraction and analysis method based on multi-region gated fusion, comprising: Step S1: Generate a target region mask from the image to be analyzed using a pre-trained region extraction model (such as semantic segmentation or saliency detection model). Decompose the image into a global image and multiple local target sub-images through mask operations (such as Hadamard product). Construct hierarchical input data containing prior knowledge of the target region structure. The specific steps include: performing preprocessing operations on the image to be analyzed, including size normalization to 224×224 pixels, pixel value normalization, and data augmentation; using a pre-trained lightweight region extraction model to generate a target region mask through an encoder-decoder structure; and performing Hadamard product operation on the mask and the original image to decompose the image into a global image and multiple local sub-images, thereby constructing hierarchical input data containing prior information about the target region structure for subsequent feature extraction.

[0025] Specifically, the image to be analyzed is input into a pre-trained semantic segmentation network to identify the boundaries of key target regions (such as foreground objects and structural regions) in the image, generating binary region masks and background masks corresponding to different structural regions. The image to be analyzed is then subjected to pixel-level Hadamard product operations with each of the region masks to generate local region sub-images that retain only pixels of specific target regions, while the original image is retained as the input to the global branch. The global image (i.e., the original image) and multiple local region sub-images constitute parallel hierarchical input data, thereby eliminating noise interference from irrelevant background regions and providing clear spatial structural constraints for subsequent feature extraction.

[0026] Step S2: Input the global image and multiple local target sub-images into a visual feature extractor with shared weights (CNN or ViT visual feature extractor) to perform depth feature extraction, and generate global patch features and local patch features corresponding to each region respectively; The specific steps include: inputting the global image and multiple local target sub-images into a visual feature extractor with shared weights to perform deep feature extraction operations. The visual feature extractor uses the convolutional layer of a pre-trained model as the feature encoding backbone network, removes the fully connected classification layer of the original model and adds an adaptive average pooling layer to uniformly aggregate the feature maps into a fixed-dimensional feature vector; extracting global patch features containing the overall anatomical structure from the global image by sharing network parameters, and extracting local patch features focused on specific target regions from the local sub-images respectively.

[0027] Specifically, by utilizing the parameter sharing mechanism of the visual feature extractor, the same feature mapping operation is performed on the input global image and each local region sub-image, projecting image data from different perspectives onto the same high-dimensional semantic feature space, thereby generating global patch features containing rich semantic information and local patch features corresponding to each target structure.

[0028] Step S3: Input the global tile features into the dual attention module, and process them in parallel through the location and channel attention branches to capture long-range spatial dependencies and channel semantic associations, thereby obtaining global enhanced features; The specific steps include: inputting global tile features into a dual attention module; calculating the product of the input feature matrix and its transpose through the positional attention branch; standardizing the matrix using a scaling factor; and generating a spatial attention map using the softmax function. The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix. The two attention branches are then integrated using a weighted fusion strategy. The spatial attention map captures the correlations between different spatial locations within the global tile markers, enhancing the feature representation of important regions. The channel attention map (CAG) models the dependencies between different feature channels within the global tile markers, highlighting feature channels important to the current task. The integration of the two attention branches using a weighted fusion strategy achieves synergistic enhancement of features in both the spatial and channel dimensions.

[0029] Step S4: Employ a regional gating fusion mechanism, where the regional gating network calculates the dynamic weight coefficients of global enhanced features and local features, and then weights and fuses the two to generate a regional fusion feature that takes into account both global and local aspects. The specific steps include: the regional gating fusion mechanism comprises multiple independent channel-level gating fusion modules, each corresponding to a local target region; for the k-th local region, the global enhanced features are... With the kth local feature Average pooling is performed on the tile dimension M to generate global summary features. Local summary features Where B is the batch size and D is the feature dimension; the two feature sums are added together and then input into a multilayer perceptron (MLP) structure to generate channel-gated weights. The MLP uses a two-layer fully connected network and the gating weights are activated by a sigmoid activation function. The value range is limited to between 0 and 1; channel gating weight After broadcasting to the tile dimension via the dimension expansion operation, we get... Through the soft fusion formula: ; The global enhanced features and the k-th local features are fused through a channel-wise weighted fusion process to generate a region fusion feature that takes into account both global semantic context and local anatomical structure. In the formula, This represents the Hada code product.

[0030] Specifically, global average pooling is performed on both the global enhanced feature sequence and the local tile region feature sequence to compress the semantic information of the spatial dimension into compact summary features, thereby capturing the overall distribution characteristics of different feature branches at the channel level. The global summary features and local summary features are summed element-wise and projected onto the hidden feature space through a multilayer perceptron (MLP). The complex nonlinear dependencies between channels are modeled using the sigmoid activation function to generate channel gating weights g that can adaptively measure feature importance. k Based on this gated weight, a complementary moving average fusion mechanism is constructed, utilizing the weight g. k and its complementary value 1-g k The global and local features are dynamically weighted and recombined separately. Through feature recalibration, an adaptive balance between global context and local details is achieved, generating a region fusion feature F that acts as an expert for a specific target region. k (k=1, 2, 3, 4).

[0031] Step S5: The multi-region fusion feature is input into the hybrid expert style aggregation module, and after learning and weighting, a unified fusion visual feature is generated and sent to the Transformer encoder for encoding to obtain a high-order semantic feature representation; The specific steps include: multiple sets of region fusion features are input into a hybrid expert style aggregation module for multi-region semantic integration. The hybrid expert style aggregation module treats each region fusion feature as an independent expert and adaptively calculates the contribution weight of each region to the final diagnosis through a learnable routing network; for each region fusion feature... Average pooling is performed at the tile dimension to obtain region vector representations. Each region vector is then input into an independent linear projection layer to map it into a route score scalar. Route weights are generated using the Softmax normalization function. Extending to the tile dimension yields Using the weighted summation formula: ; Adaptive aggregation of regional fusion features to generate unified fused visual features , in the formula, This represents element-wise multiplication. The unified fused visual features are mapped to a unified hidden dimension space through a linear projection layer and then input into a multi-layer Transformer encoder for deep semantic encoding. Each Transformer encoder layer contains a multi-head self-attention module, residual connections, layer normalization, and a feedforward neural network to generate encoder semantic features containing high-order global semantic information.

[0032] Specifically, the mean of the fused feature Fk of each region is encoded through a linear transformation, and then the region-level weighting coefficients a are learned using the Softmax function.k Based on this coefficient, the fusion features of each region are weighted and summed to obtain a unified fused visual label. Then, the fused visual label is fed into a Transformer encoder that includes a multi-layer self-attention mechanism and a feedforward network (FFN). The self-attention mechanism captures the semantic associations within the visual label, and the layer normalization and residual connections are combined to enhance the feature expression capability. After multi-layer encoding, the output generates the encoder semantic features to support the subsequent decoding process.

[0033] Step S6: The guided query tensor generated by the memory guidance module through the unified fusion visual features is used as the query input of the cross-attention module in the structured decoder, and interacts with the semantic features output by the encoder in a non-autoregressive manner to output the target query features. The specific steps include: feeding the unified and fused visual features into the memory guidance module, obtaining the global visual representation through average pooling, inputting it into a multilayer perceptron for feature encoding, generating a modulation parameter vector through a fully connected layer and splitting it into scaling parameters and bias parameters, and conditionally modulating the query tensor to obtain the guidance query tensor; in the structured decoding stage, the guidance query tensor serves as the query for the first layer of the structured decoder's multi-head cross-attention, and the semantic features output by the encoder serve as the key and value; the structured decoder consists of multiple stacked decoding layers, each containing multi-head cross-attention, layer normalization, feedforward network, and Dropout module, removing the multi-head self-attention structure in the traditional Transformer decoder, and using a parallel non-autoregressive decoding method to model the correspondence between the target query and image semantic features, and outputting the target query features.

[0034] Specifically, the unified and fused visual features are fed into the memory guidance module, where a compact global visual representation is first obtained through average pooling. This global summary feature is then input into a multilayer perceptron (MLP) for feature encoding and mapped to generate a set of modulation parameter vectors. This vector is further decomposed into scaling parameters. and bias parameters The initial query tensor is conditionally modulated to obtain a guided query tensor guided by global visual information. In the structured decoding stage, the disease query guidance vector generated by the memory guidance module serves as the query (Q) of the first layer of the structured decoder's multi-head cross-attention, while the semantic features output by the Transformer encoder serve as the key (K) and value (V). The structured decoder consists of N stacked decoding layers, each containing multi-head cross-attention, layer normalization, a feedforward network, and a Dropout module. Only the first layer uses the guided query tensor as the input query; subsequent layers use the output of the previous layer's decoder as the query vector. Furthermore, this decoder removes the multi-head self-attention structure from the traditional Transformer decoder, employing a parallel non-autoregressive decoding approach to directly model the correspondence between the target query and image semantic features, and refines the target query representation layer by layer.

[0035] Step S7: The target query features output by the structured decoder are fed into multiple parallel prediction heads. Each prediction head shares the same input features but performs inference independently, while simultaneously completing multi-dimensional structured information prediction and outputting structured analysis results.

[0036] The specific steps include: the structured decoder outputs the target query features and feeds them into a multi-task prediction head, which then performs multi-dimensional structured information prediction through parallel sub-networks, and outputs the structured analysis results in parallel; the system uses a multi-task joint loss function for end-to-end training, including target category recognition, existence judgment, target confidence estimation, attribute state classification, and target location localization, and finally outputs the structured analysis results in parallel; the joint loss calculation formula is: ; Where N is the total number of prediction heads. Let be the loss function for the i-th task. Let be the loss weight coefficient for the i-th task. The optimal balance is determined through hyperparameter tuning, and the specific form of the loss function for each task is flexibly designed according to the task characteristics. In the subsequent result decoding stage, the system provides an optional conditional filtering mechanism. This mechanism supports flexibly configuring any combination of tasks as preconditions based on task characteristics. When the conditional filtering mechanism is enabled, a binary mask M(k) is introduced for each target k in the decoding stage, defined as: ; in This represents the set of prerequisite tasks. This indicates an indicator function that takes the value 1 when the condition is met and 0 otherwise. The preset threshold is the threshold value for the j-th preceding task.

[0037] To better understand the present invention, the following embodiment illustrates steps S1-S7 of the image feature extraction and analysis method based on multi-region gated fusion of the present invention, as follows: Step S1: The image to be analyzed is preprocessed, including adjusting the image size to 224×224 pixels, normalizing the pixel values, and introducing data augmentation operations such as random horizontal flipping, random rotation, brightness and contrast adjustment during the training phase. Then, the preprocessed image is input into a pre-trained lightweight region extraction model (such as the Unet series semantic segmentation model) to generate a corresponding target region mask. The region extraction model achieves fine segmentation of key target regions through an encoder-decoder structure. Based on this, a Hadamard product operation is performed between the target region mask (binary mask) and the original image. The pixels in the original image corresponding to the pixel positions with a value of 1 in the binary mask are retained, while the pixel positions with a value of 0 are suppressed to zero. This decomposes the image into a global image that retains the overall semantic information and multiple local sub-images that only contain the corresponding target region information, thereby constructing hierarchical input data containing prior information about the target region structure for subsequent feature extraction and semantic modeling processes.

[0038] Step S2: The global image obtained in Step S1 and four local region sub-images (a total of five images) are simultaneously input into a shared-weight visual feature extractor to perform a unified deep feature extraction operation. The visual feature extractor can be a convolutional neural network (CNN) (such as ResNet101, VGG, or other pre-trained models) or a visual Transformer (ViT) structure. The front-end convolutional layer of the pre-trained model is used as the feature encoding backbone network. The last two fully connected classification layers of the original model are removed, and a 7×7 adaptive average pooling layer is added at the end of the feature extraction to uniformly aggregate feature maps of different spatial sizes into a fixed-dimensional feature vector. Its network parameters remain consistent when processing the global image and each local sub-image, thereby ensuring that images of different scales and different target regions are represented and learned in the same feature space. During the feature extraction process, the visual feature extractor performs parallel hierarchical feature encoding on the five input images. It extracts global patch features (with a shape of B×M×D, where B is the batch size, M is the number of patches, and D is the feature dimension) that contain the overall target structure and contextual semantics from the global image. It also extracts local patch features (also with a shape of B×M×D) that focus on specific target regions from the four local sub-images. These features represent the macroscopic semantic information of the overall image and the local structure and detail information of each target region. The feature extraction mechanism with shared weights ensures the comparability and consistency of the global and local views in the feature space, avoiding the feature distribution offset problem caused by using independent extractors. This provides a consistent and alignable feature representation foundation for subsequent cross-regional semantic fusion, multi-scale feature alignment, and structured analysis modeling.

[0039] Step S3: As Figure 3 As shown, the global tile features (shape B×M×D) obtained in step S2 are input into the dual attention enhancement module for feature optimization. This module uses a parallel processing mechanism of positional attention branch and channel attention branch to semantically enhance the global tile features from the spatial dimension and channel dimension, respectively, to capture the long-range dependencies between different spatial locations in the image and the semantic correlations between different feature channels; the positional attention branch targets the input feature matrix. (Where B is the batch size, N is the number of tiles, and C is the number of channels) Calculate the spatial attention map (SAG), first through the feature matrix. Rather than transpose Matrix product after scaling factor After standardization, a softmax normalization function is applied, and the calculation formula is as follows: ; This spatial attention map can quantify and enhance the correlation weights between different spatial locations in an image, enabling the model to focus on combinations of regions that are strongly correlated in terms of object geometry. The output of the location attention branch is as follows: ; in These are learnable weight parameters used to control the strength of spatial attention enhancement, and residual connectivity terms. Preserving original feature information to prevent gradient vanishing; the channel attention branch calculates the transpose of the feature matrix. With the original matrix The product of the channel attention map (CAG) is generated by the following formula: ; This attention map can model the dependencies between different feature channels, adaptively highlighting feature channels that are discriminative for the current structured analysis task, and suppressing the responses of redundant or noisy channels. The output of the channel attention branch is as follows: ; in The weights are learnable parameters used to adjust the degree of influence of channel attention; the two attention branches are integrated through a weighted fusion strategy, and the final global enhanced feature calculation formula is as follows: ; in To integrate weight parameters, in actual implementation, the parameters... and Initialize to 0.1 to ensure stability during the initial training phase, and then fuse the weights. Setting it to 0.5 balances the contributions of spatial and channel dimension features, achieving synergistic enhancement of spatial long-range dependence and channel semantic association, thereby generating a globally enhanced feature representation that simultaneously contains rich spatial structural information and discriminative channel features, providing a more expressive visual feature foundation for subsequent structured analysis.

[0040] Step S4: As Figure 4 As shown, the region-gated fusion mechanism comprises four independent channel-level gated fusion modules, each corresponding to one of the four local target regions. Each module can learn an adaptive fusion strategy for a specific region. For the k-th local region, the fusion process first applies global enhanced features. With the kth local feature Average pooling is performed on the tile dimension M to generate global summary features. Local summary features The calculation formula is: ; ; Where B is the batch size and D is the feature dimension; then the two summary features are added and input into a multilayer perceptron (MLP) structure. The MLP uses a two-layer fully connected network. The first layer expands the input from D dimensions to 2D dimensions and passes it through the ReLU activation function to capture complex nonlinear patterns. The second layer compresses the features back to D dimensions and generates channel-gated weights through the Sigmoid activation function. The calculation formula is: ; Among them, , , This is the Sigmoid activation function. The gate weights range from 0 to 1 and are used to quantify the dependence of each feature channel on global and local features; channel gate weights. After broadcasting to the tile dimension via the dimension expansion operation, we get... Ultimately, the soft fusion formula was used: ; The global enhanced feature and the k-th local feature are fused through a channel-wise weighted fusion, where This represents the Hada code product, i.e., element-wise multiplication. When the gate weight is close to 1, it indicates that the channel relies more on the semantic guidance of global features; when the gate weight is close to 0, it indicates that the channel relies more on the target details of local features, thereby achieving adaptive adjustment of the feature fusion strategy. The above gate fusion operation is performed on the four local regions respectively, generating four sets of region fusion features that take into account both global semantic context and local target structure. This provides a feature representation that retains global consistency while highlighting local specificity for subsequent multi-region semantic aggregation and structured analysis modeling.

[0041] Step S5: As Figure 5 As shown, the four sets of region fusion features generated in step S4 are... The input is fed into a hybrid expert-style aggregation module for multi-region semantic integration. This module treats each region's fused feature as an independent "expert," adaptively calculating the contribution weight of each region to the final structured analysis through a learnable routing network, thus achieving content-aware dynamic feature aggregation. The hybrid expert-style aggregation module first processes the fused features of each region... Perform average pooling on the tile dimension M to obtain the vector representation of the k-th region. The calculation formula is: ; This vector encapsulates the overall semantic information of the region; subsequently, each region vector is... The inputs are fed into four independent linear projection layers, each mapping a D-dimensional region vector to a single route scoring scalar. The mapping process is represented as: ; in and These are the weights and bias parameters of the k-th linear projection layer. This linear projection layer has the ability to map region semantics to importance scores; it concatenates the four route scores into a score vector. Routing weights are generated using the Softmax normalization function. The calculation formula is: ; in Satisfy constraints and The routing weights reflect the relative importance of each target region in the current input image to the structured analysis task. For example, for an image containing key targets, the weight of the main target region is... The routing weight may be significantly higher than in other areas; Extending to the tile dimension yields By using the weighted summation formula ( (This represents element-wise multiplication) ; Adaptive aggregation of the fusion features from the four regions is performed to generate a unified fused visual feature. This feature simultaneously includes global semantic context, local target details, and content-based weighted information on regional importance.

[0042] The unified fusion visual features Then, it is mapped to a unified hidden dimension space of D dimensions (default is 512) through a linear projection layer to ensure the consistency of feature dimensions. The projection process is represented as follows: ; in For the unified fusion of visual features as input, and The projection layer parameters are then input into a multi-layer Transformer encoder (3 layers in total) for deep semantic encoding. Each encoder layer includes a multi-head self-attention module, residual connections, layer normalization, and a feedforward neural network. The encoding process progressively improves the abstraction level and semantic richness of the feature representation, ultimately generating encoder semantic features containing high-order global semantic information. This serves as the core visual feature representation for subsequent structured analysis.

[0043] Step S6: As Figure 6As shown, the memory guidance module receives the unified fused visual features output in step S5. As input, the feature tensor is first subjected to average pooling along the sequence dimension, compressing the M feature vectors into a single global visual summarization feature. The calculation formula is: ; Subsequently, this global visual summarization feature The input is directly fed into a multilayer perceptron (MLP) for nonlinear transformation to generate a modulation parameter vector. The calculation process is expressed as follows: ; in The weight parameters of the MLP, This is the bias parameter. The parameter vector is split to generate two branches, which respectively form the scaling factor. and offset factor Meanwhile, the module maintains a set of updatable initial query tensors. This tensor is initialized only once during model creation (using a Xavier uniform distribution), and is not regenerated during each forward propagation. Afterward, this tensor persists in the network as a trainable parameter of the model, continuously updating and evolving through backpropagation throughout the training process. Its status is equivalent to the weight parameters of convolutional layers, and each query vector can adaptively learn to capture the abstract semantic feature patterns of the corresponding target category. During forward propagation, the initialized query tensor is first expanded to the batch dimension through a broadcast mechanism. Subsequently, it interacts with the modulation parameters generated from the global visual summarization features, and dynamically injects visual information through an affine transformation mechanism. The calculation formula is as follows: ; in This represents element-wise multiplication, an operation that fuses global visual context information from the current input image into a learnable query vector. This allows the query representation to adaptively adjust based on the specific input content, ultimately generating a guided query tensor that integrates global visual information and domain prior knowledge. .

[0044] like Figure 7 As shown, the guiding query tensor As the initial query input for the first-layer cross-attention module of the structured decoder, the dimensions are first aligned using linear projection to obtain... Simultaneously, the encoder semantic features generated in step S5 are... As key and value inputs, the attention weight distribution between the query and encoder features is calculated through a cross-attention mechanism, capturing the correlation strength between each disease query and features of different spatial regions of the image. The feature representations of K target categories are decoded in parallel in a non-autoregressive manner, avoiding the sequence dependency and inference latency of traditional autoregressive decoding. The decoder adopts a multi-layer stacked structure with L layers. Each layer sequentially includes a multi-head cross-attention layer, layer normalization, a feedforward neural network, layer normalization, and a Dropout regularization module. For the , ... layer( First, calculate the output of multi-head cross-attention: ; in For the number of attention heads, The dimension is defined for each head; then residual connections and layer normalization are performed: ; Next, a nonlinear transformation is performed using a feedforward neural network: ; in , , Residual connections and layer normalization are applied again, and finally the output of this layer is obtained through Dropout: ; The cross-attention modules in layers 2 through L sequentially use the disease query features output from the previous layer as the query input for the current layer, forming a hierarchical feature refinement mechanism that encoders semantic features. Throughout all L layers, the key and value remain constant, ensuring that each layer can extract information from the original visual features. This iterative approach continuously refines the discriminative power and semantic richness of the disease feature representation, ultimately resulting in the Lth layer outputting target query features containing rich diagnostic semantic information. .

[0045] Step S7: As Figure 8 As shown, the target query features Simultaneously, the data is input to N independent task prediction heads (N=5 in this implementation). Each prediction head has a dedicated network structure designed for a specific structured analysis task, enabling parallel prediction and joint optimization across multiple analysis dimensions. The output of the i-th prediction head is denoted as... ,in This represents the output dimension of the i-th task (for binary classification tasks). For multi-classification tasks (Number of categories). Specifically, in this implementation, the first two prediction heads use a single-layer fully connected network and a Sigmoid activation function, respectively outputting... (Probability of target mention) and (Target confirmation probability); The third prediction head uses a single-layer fully connected network and the Softmax activation function, outputting... The third dimension corresponds to three confidence levels, satisfying the normalization constraint. The fourth prediction head also uses a single-layer fully connected network and the Softmax activation function, and the output... The third dimension corresponds to four degree levels, used to evaluate the target's level attributes; the fifth prediction head adopts a multi-branch parallel structure, with each branch containing an independent fully connected layer and a Softmax activation function, maintaining a dedicated attribute vocabulary for the k-th target, and outputting... ,in This represents the maximum capacity of the vocabulary for all target attributes; It should be noted that, to ensure parallelization efficiency, the aforementioned N prediction heads will calculate all prediction information for all targets at once during the forward propagation process. In the subsequent result decoding stage, the system provides an optional conditional filtering mechanism: when this mechanism is enabled (via hyperparameters)... (Control), in the decoding stage, a binary mask M(k) is introduced for each target k, which is defined as ; in This represents the set of prerequisite tasks. This indicates an indicator function (it takes the value 1 when the condition is met, and 0 otherwise). This is a preset threshold for the j-th prerequisite task. The framework supports flexible configuration of any combination of tasks as prerequisites; in this implementation, it adopts... The configuration is as follows: ; in These correspond to the thresholds for the mentioned conditions and the confirmation conditions, respectively. A target is considered considered only if both prerequisite conditions are met. (Conditions mentioned) and (Confirmation conditions) where For a preset threshold, this target is included in other prediction heads ( Only the output of ) is considered valid information and included in the final structured output; if the objective satisfies but If so, only the negative result of the target is output, without needing to decode subsequent fine-grained attribute information. If the objective is not met. If the target is not specified, then the information in all prediction heads is completely ignored; when this mechanism is disabled ( All targets are directly incorporated into the structured output of all prediction heads without any conditional filtering.

[0046] Taking the task of generating medical image reports as an example, among which = (Whether a certain disease was mentioned) = (Whether a disease is negative or positive), only if a disease is successfully identified as "mentioned" ( ,in (Referring to the threshold) and judged as "positive" ( ,in When the threshold for a positive result is reached, the confidence estimate, severity classification, and lesion location identification results for the disease are considered valid information and included in the final structured report output; if the disease is mentioned but judged as "negative" (where the threshold is not met), the results are considered valid information. If the report only explicitly states that the disease was a negative finding, there is no need to decode subsequent fine-grained attribute information; if the disease is not mentioned ( If the prediction information for the disease is completely ignored through a masking operation and not included in any text generation process, then invalid predictions are effectively filtered out and the logical rationality of the output results is ensured. All prediction heads are jointly optimized through a multi-task learning framework, and the total loss function adopts a weighted summation form. ; Where N is the total number of prediction heads. Let be the loss weight coefficient for the i-th task. Let be the loss function for the i-th task. By adjusting the weight coefficients, the relative importance of each task in joint training can be flexibly controlled, achieving a balance in multi-objective optimization. The specific form of the loss function for each task is flexibly designed according to the task characteristics. For binary classification tasks with class imbalance problems (such as preconditioning tasks), Focal Loss can be used. ; in For the target total number, For the first The first goal in Real labels on each task For the corresponding predicted probability, To balance the weights for each category, The focusing parameter. This loss function is based on the modulation factor. Reducing the loss weights for easily separable samples allows the model to focus on optimizing difficult-to-separate samples, effectively alleviating the problem of imbalanced training data distribution. Taking the medical image report generation task as an example, the total loss function is specified as follows: ; The specific forms of the loss functions for each task include the Focal Loss form for disease identification loss: ; in For the true label of whether the k-th disease was mentioned, For the corresponding predicted probability, To balance the weights for each category (default setting is 0.25), The focus parameter (default setting is 2.0), weights ; The loss for negative / positive judgment is expressed as a binary cross-entropy: ; in, As a true label for whether the disease is positive, weight ; The confidence level estimation loss is expressed using the multi-class cross-entropy form: ; in, For one-hot encoded true labels (C=3, corresponding to low / medium / high confidence levels), the weights are... ; The severity grading loss also adopts the multi-class cross-entropy form: ; in, The true severity labels for one-hot encoding (S=4, corresponding to unknown / mild / moderate / severe), weights ; The location loss is calculated using multi-class cross-entropy. ; in The true label of the target location for one-hot encoding ( (Number of location categories), weight .

[0047] End-to-end training enables the model to achieve optimal performance across multiple prediction dimensions simultaneously, and the outputs of each prediction head collectively constitute a complete set of structured prediction results. ; in Indicates the first The prediction output of each prediction head for all targets. This represents the total number of prediction heads. The joint optimization strategy ensures coordination between predictions across dimensions, avoiding inconsistencies that might arise from individual training. After conditional masking filtering, the system transforms the prediction results for each dimension into a more readable, structured output using a predefined mapping table. Specifically, each target that passes the filter... Predictive information It will be mapped to the corresponding set of semantic tags. ,in Indicates the first Each prediction head outputs its corresponding item in a predefined vocabulary. These multi-dimensional labels are assembled according to a preset output template to generate a structured description that conforms to domain specifications. The output template defines the order, connection method, and format rules of the information in each dimension, ensuring the consistency and readability of the final output. The entire generation process follows a pipeline architecture of "joint prediction → filtering → mapping → assembly," providing accurate and concise multi-dimensional information support for downstream applications.

[0048] Taking the medical image report generation task as an example, end-to-end training enables the model to achieve optimal performance simultaneously across five dimensions: disease identification, negative / positive judgment, confidence estimation, severity grading, and location localization, ultimately forming a structured set of diagnostic results. ; After conditional masking filtering, the prediction results of each dimension are converted into standardized medical terms (such as "pneumonia mentioned" + "positive" + "medium confidence" + "mild" + "right upper lobe") through a predefined clinical terminology mapping. Then, a structured text diagnostic report can be generated by assembling the report according to the clinical report writing specifications.

[0049] This invention presents an image feature extraction and analysis method based on multi-region gating fusion. In a dual-attention module, parallel processing of positional and channel attention branches enables the simultaneous capture of long-range spatial dependencies and channel semantic associations in the global image, effectively enhancing the expressive power of global features. In the region gating fusion mechanism, the gating network adaptively adjusts the weights of global enhanced features and local detail features, overcoming the problem of easy loss of local fine-grained features or separation from the global context in traditional methods, achieving organic complementarity of multi-scale features. In the memory-guided and decoding stages, a guiding query tensor generated by the memory-guided module is introduced as a query for the cross-attention module in the structured decoder, providing accurate prior information guidance for the decoding stage. Combined with a non-autoregressive decoding mode, it breaks the dependency limitations of traditional serial generation. Combined with a multi-task prediction head, it can output multi-dimensional structured analysis results in parallel. This technical solution fully utilizes prior knowledge of the target region structure in the image, significantly improving the feature extraction accuracy and inference analysis efficiency of fine-grained targets (such as medical lesions and industrial defects) in various complex images, providing high-accuracy and high-real-time technical support for the automated analysis of complex visual scenes. This invention addresses the problems in existing image refinement analysis, such as insufficient attention to local fine-grained targets, imbalance between global and local feature fusion, and lack of structured semantic description in the output results. This method effectively integrates prior knowledge of the target region's structure with global visual information, and resolves the imbalance between local and global features through a gating mechanism, thereby significantly improving the fine-grained recognition capability and the accuracy of structured analysis results in complex scenes.

[0050] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A method for image feature extraction and analysis based on multi-region gated fusion, characterized in that, include: The image to be analyzed is used to generate a target region mask using a pre-trained region extraction model. The mask is then decomposed into a global image and multiple local target sub-images to construct hierarchical input data containing prior knowledge of the target region structure. The global image and multiple local target sub-images are input into a visual feature extractor with shared weights to perform depth feature extraction, generating global patch features and local patch features corresponding to each region respectively. The global tile features are input into the dual attention module and processed in parallel by the location and channel attention branches to capture long-range spatial dependencies and channel semantic associations, thus obtaining global enhanced features. A regional gating fusion mechanism is adopted, in which the regional gating network calculates the dynamic weight coefficients of global enhanced features and local features, and the two are weighted and fused to generate regional fusion features that take into account both global and local features. Multi-region fusion features are input into the hybrid expert style aggregation module, which learns and weights them to generate unified fusion visual features, which are then fed into the Transformer encoder for encoding to obtain high-order semantic feature representations; The guided query tensor generated by the memory guidance module through the unified fusion of visual features is used as the query input of the cross-attention module in the structured decoder. It interacts non-autoregressively with the semantic features output by the encoder to output the target query features. The target query features output by the structured decoder are fed into multiple parallel prediction heads. Each prediction head shares the same input features but performs inference independently, thereby completing multi-dimensional structured information prediction and outputting structured analysis results.

2. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 1, characterized in that, The specific steps for constructing hierarchical input data containing prior knowledge of the target region structure include: First, using a pre-trained region extraction model to generate a target region mask from the image to be analyzed. This mask is then used to decompose the image into a global image and multiple local target sub-images. The image to be analyzed undergoes preprocessing operations including size normalization to 224×224 pixels, pixel value normalization, and data augmentation. A pre-trained lightweight region extraction model is used to generate a target region mask through an encoder-decoder structure. Based on the mask and the original image, a Hadamard product operation is performed to decompose the image into a global image and multiple local sub-images, thereby constructing hierarchical input data containing prior information about the target region structure for subsequent feature extraction.

3. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 1, characterized in that, The specific steps for inputting the global image and multiple local target sub-images into a shared-weight visual feature extractor to perform depth feature extraction and generate global patch features and local patch features corresponding to each region include: The global image and multiple local target sub-images are input into a visual feature extractor with shared weights to perform deep feature extraction. The visual feature extractor uses the convolutional layer of the pre-trained model as the feature encoding backbone network, removes the fully connected classification layer of the original model and adds an adaptive average pooling layer to aggregate the feature maps into a fixed-dimensional feature vector. By sharing network parameters, global patch features containing the overall anatomical structure are extracted from the global image, and local patch features focusing on specific target regions are extracted from the local sub-images.

4. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 1, characterized in that, The specific steps for inputting global tile features into a dual attention module, and processing them in parallel through location and channel attention branches to capture long-range spatial dependencies and channel semantic associations, resulting in global enhanced features, include: Global tile features are input into the dual attention module. The positional attention branch calculates the product of the input feature matrix and its transpose, performs normalization by scaling factor, and applies the softmax function to generate a spatial attention map. The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix. The two attention branches are integrated through a weighted fusion strategy.

5. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 1, characterized in that, The specific steps of employing a regional gating fusion mechanism, which involves calculating dynamic weight coefficients for global and local enhanced features using a regional gating network, and then weighting and fusing the two to generate regional fused features that take into account both global and local aspects, include: The regional gating fusion mechanism comprises multiple independent channel-level gating fusion modules, each corresponding to a local target region; for the k-th local region, global enhancement features are applied. With the kth local feature Average pooling is performed on the tile dimension M to generate global summary features. Local summary features Where B is the batch size and D is the feature dimension; the two feature sums are added together and then input into a multilayer perceptron (MLP) structure to generate channel-gated weights. The MLP uses a two-layer fully connected network and the gating weights are activated by a sigmoid activation function. The value range is limited to between 0 and 1; channel gating weight After broadcasting to the tile dimension via the dimension expansion operation, we get... Through the soft fusion formula: ； The global enhanced features and the k-th local features are fused through a channel-wise weighted fusion process to generate a region fusion feature that takes into account both global semantic context and local anatomical structure. In the formula, This represents the Hada code product.

6. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 5, characterized in that, The specific steps involved in inputting multi-region fusion features into a hybrid expert style aggregation module, learning and weighting to generate unified fused visual features, and then feeding them into a Transformer encoder to obtain high-order semantic feature representations include: Multiple sets of region fusion features are input into a hybrid expert-style aggregation module for multi-region semantic integration. This module treats each region fusion feature as an independent expert and adaptively calculates the contribution weight of each region to the final diagnosis through a learnable routing network. Average pooling is performed at the tile dimension to obtain region vector representations. Each region vector is then input into an independent linear projection layer to map it into a route score scalar. Route weights are generated using the Softmax normalization function. Extending to the tile dimension yields Using the weighted summation formula: ； Adaptive aggregation of regional fusion features to generate unified fused visual features , in the formula, This represents element-wise multiplication. The unified fused visual features are mapped to a unified hidden dimension space through a linear projection layer and then input into a multi-layer Transformer encoder for deep semantic encoding. Each Transformer encoder layer contains a multi-head self-attention module, residual connections, layer normalization, and a feedforward neural network to generate encoder semantic features containing high-order global semantic information.

7. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 1, characterized in that, The specific steps for using the guided query tensor generated by the memory-guided module through the unified fusion of visual features as the query input to the cross-attention module in the structured decoder, and interacting non-autoregressively with the semantic features output by the encoder to output the target query features include: The unified and fused visual features are fed into the memory guidance module. After obtaining the global visual representation through average pooling, it is input into a multilayer perceptron for feature encoding. The vector of modulation parameters is generated by mapping through a fully connected layer and then split into scaling parameters and bias parameters. The query tensor is conditionally modulated to obtain the guidance query tensor. In the structured decoding stage, the guidance query tensor serves as the query for the first layer of the structured decoder with multi-head cross-attention. The semantic features output by the encoder serve as the key and value. The structured decoder consists of multiple stacked decoding layers. Each layer contains multi-head cross-attention, layer normalization, feedforward network and Dropout module. The multi-head self-attention structure in the traditional Transformer decoder is removed. A parallel non-autoregressive decoding method is used to model the correspondence between the target query and the image semantic features and output the target query features.

8. The image feature extraction and analysis method based on multi-region gated fusion as described in claim 1, characterized in that, The specific steps involved in feeding the target query features output from the structured decoder into multiple parallel prediction heads, each sharing the same input features but performing inference independently, simultaneously completing multi-dimensional structured information prediction and outputting structured analysis results, include: The structured decoder outputs target query features and feeds them into a multi-task prediction head. Parallel sub-networks then perform multi-dimensional structured information prediction, outputting structured analysis results in parallel. The system employs a multi-task joint loss function for end-to-end training, including target category identification, existence judgment, target confidence estimation, attribute state classification, and target location localization. Finally, it outputs structured analysis results in parallel. The joint loss calculation formula is as follows: ； Where N is the total number of prediction heads. Let be the loss function for the i-th task. Let be the loss weight coefficient for the i-th task. The optimal balance is determined through hyperparameter tuning, and the specific form of the loss function for each task is flexibly designed according to the task characteristics. In the subsequent result decoding stage, the system provides an optional conditional filtering mechanism. This mechanism supports flexibly configuring any combination of tasks as preconditions based on task characteristics. When the conditional filtering mechanism is enabled, a binary mask M(k) is introduced for each target k in the decoding stage, defined as: ； in This represents the set of prerequisite tasks. This indicates an indicator function that takes the value 1 when the condition is met and 0 otherwise. The preset threshold is the threshold value for the j-th preceding task.

Images