Adaptive feature fusion method and device based on entropy difference gate, equipment, medium and product

Abstract

This disclosure relates to an adaptive feature fusion method, apparatus, device, medium, and product based on entropy difference gating. The method targets any network layer in the encoder-decoder architecture network decoding process. It calculates the entropy difference between the global features output from the upsampled network layer and the local features input from the skip connections, quantifying the deterministic differences in feature representation across the spatiotemporal and channel dimensions. Then, using the quantized deterministic difference as a driving signal, it adaptively generates local gating weights corresponding to the local features and global gating weights corresponding to the global features. The local and global gating weights are then used to perform weighted modulation and fusion of the corresponding local and global features, effectively suppressing the interference of low-deterministic features on the fusion result and improving the accuracy and robustness of multi-scale feature fusion.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and in particular to an adaptive feature fusion method, apparatus, device, medium, and product based on entropy difference gating. Background Technology

[0002] Encoder-decoder based neural networks (referred to as encoder-decoder architecture networks) are widely used in various dense prediction tasks, such as image segmentation and speech separation. In this network architecture, the effectiveness of the decoding stage depends on the network layers' ability to integrate multi-scale information.

[0003] In related technologies, the network layers used for decoding in encoder-decoder architectures typically employ two types of fusion strategies: simple fusion methods, such as element-wise addition or concatenation; and weighted methods based on attention mechanisms. These fusion strategies integrate multi-scale feature information before outputting the result.

[0004] However, simple fusion methods fail to distinguish the importance of features at different scales, making the integrated output features susceptible to interference from low-quality redundant information, thus weakening the feature representation ability. In scenarios with blurred target boundaries or complex backgrounds, this can easily lead to coarse segmentation edges or missegmentation. Meanwhile, attention-based weighting typically applies to features at a single scale, mainly capturing the contextual dependencies within that scale, but lacks explicit modeling of cross-scale feature differences. This results in feature alignment bias and semantic ambiguity in regions with complementary information at multiple scales. Summary of the Invention

[0005] In view of this, this disclosure proposes an adaptive feature fusion method, apparatus, device, medium, and product based on entropy difference gating. This method is applicable to the decoding stage of encoder-decoder network structures. For any network layer used to implement decoding, by quantifying the difference in feature representation determinism between the global features upsampled at that network layer and the local features input via skip connections, an adaptive gating weight adapted to both local and global features is generated. Based on this gating weight, global and local features are modulated, effectively suppressing the interference of low-determinism features on the fusion result, improving the accuracy of multi-scale feature fusion in the decoding stage, and thus enhancing the overall feature representation capability of the network.

[0006] According to one aspect of this disclosure, an adaptive feature fusion method based on entropy difference gating is provided, the method being applied to the decoding stage of an encoder-decoder architecture network, the method comprising:

[0007] In the current network layer of the decoding stage, channel-dimensional feature pairs are constructed based on global and local features; wherein, the channel-dimensional feature pair includes a first feature and a second feature; the first feature is obtained by compressing the global feature channel by channel along the spatiotemporal dimension; the second feature is obtained by compressing the local feature channel by channel along the spatiotemporal dimension; the global feature is the feature output of the current network layer after upsampling; the local feature is the feature with the same spatial resolution as the current network layer in the encoding stage of the encoder-decoder architecture network;

[0008] Based on the global and local features, a spatiotemporal dimension feature pair is constructed; wherein, the spatiotemporal dimension feature pair includes a third feature and a fourth feature; the third feature is obtained by compressing the global features position by position along the channel dimension; the fourth feature is obtained by compressing the local features position by position along the channel dimension.

[0009] A first entropy difference is determined based on the channel dimension feature pair, and a second entropy difference is determined based on the spatiotemporal dimension feature pair; wherein, the first entropy difference and the second entropy difference are both determined based on the same feature comparison direction; the first entropy difference indicates the relative difference in the determinacy of feature representation between the global feature and the local feature in the channel dimension; the second entropy difference is used to indicate the relative difference in the determinacy of feature representation between the global feature and the local feature in the spatiotemporal dimension;

[0010] The first entropy difference and the second entropy difference are subjected to gating mapping to generate local gating weights and global gating weights.

[0011] Using the local and global gate weights, the global and local features in the current network layer are subjected to gated fusion processing to generate the fused features of the current network layer.

[0012] In some embodiments, the feature comparison direction includes either a global-to-local deterministic comparison direction or a local-to-global deterministic comparison direction.

[0013] The determination of the first entropy difference based on the channel dimension feature pair and the determination of the second entropy difference based on the spatiotemporal dimension feature pair include:

[0014] If the feature comparison direction is a deterministic comparison direction of the global relative to the local, then the first entropy difference is determined by the entropy increment of the first feature compared to the second feature, and the second entropy difference is determined by the entropy increment of the third feature compared to the fourth feature.

[0015] If the feature comparison direction is a deterministic comparison direction of local relative to global, then the first entropy difference is determined by the entropy increment of the second feature compared to the first feature, and the second entropy difference is determined by the entropy increment of the fourth feature compared to the third feature.

[0016] In some embodiments, the step of performing gating mapping processing on the first entropy difference and the second entropy difference to generate local gating weights and global gating weights includes:

[0017] A first gating weight is generated based on the first normalized response of the first entropy difference and the second entropy difference; the first normalized response is obtained by processing the first entropy difference and the second entropy difference respectively through a preset monotonic mapping function.

[0018] A second gating weight is generated based on the second normalized response of the first entropy difference and the second entropy difference; the second normalized response is the element-wise complementary inverse result of the first normalized response.

[0019] Based on the output monotonicity of the monotonic mapping function and the feature comparison direction, a local gate weight is determined from the first gate weight and the second gate weight, and the remaining gate weight among the first gate weight and the second gate weight is determined as the global gate weight.

[0020] In some embodiments, the current network layer includes a pre-trained joint first convolutional module and a second convolutional module; the first convolutional module includes a first convolutional layer and a third convolutional layer that are independent of each other; wherein, the first convolutional layer is used to perform convolution processing on the global features to output a first intermediate feature; the first intermediate feature is obtained by compressing the first feature along the spatiotemporal dimension channel by channel; the third convolutional layer is used to perform convolution processing on the global features to output a third intermediate feature; the third intermediate feature is obtained by compressing the third feature along the channel dimension position by position;

[0021] The second convolutional module includes a second convolutional layer and a fourth convolutional layer that are independent of each other; wherein, the second convolutional layer is used to perform convolution processing on the local features to output a second intermediate feature; the second intermediate feature is obtained by compressing the second feature along the spatiotemporal dimension channel by channel; the fourth convolutional layer is used to perform convolution processing on the local features to output a fourth intermediate feature; the fourth intermediate feature is obtained by compressing the fourth feature along the channel dimension position by position.

[0022] In some embodiments, the first convolutional module further includes a fifth convolutional layer, and the second convolutional module further includes a sixth convolutional layer;

[0023] The parameters of the fifth convolutional layer are independent of the first and third convolutional layers, and are used to perform convolution processing on the global features to output the fifth intermediate feature; the parameters of the sixth convolutional layer are independent of the second and fourth convolutional layers, and are used to perform convolution processing on the local features to output the sixth intermediate feature.

[0024] The method of using the local gate control weight and the global gate control weight to perform gated fusion processing on the global features and local features in the current network layer includes: using the global gate control weight to weight the fifth intermediate feature, and using the local gate control weight to weight the sixth intermediate feature, and fusing the weighted fifth intermediate feature and the sixth intermediate feature to generate the fused feature of the current network layer.

[0025] In some embodiments, the channel-by-channel compression along the spatiotemporal dimension includes:

[0026] For each channel of the first input feature, the information entropy value corresponding to that channel is determined based on the probability distribution corresponding to that channel; wherein, the probability distribution corresponding to that channel is used to reflect the relative importance of the first input feature at each spatiotemporal position in the feature representation of that channel; the first input feature is either the first intermediate feature or the second intermediate feature;

[0027] The information entropy values ​​of the first input feature in each channel are combined to obtain the channel-by-channel compressed features along the spatiotemporal dimension of the first input feature.

[0028] In some embodiments, the position-by-position compression along the channel dimension includes:

[0029] For each spatiotemporal location of the second input feature, the information entropy value corresponding to that spatiotemporal location is determined based on the probability distribution corresponding to that spatiotemporal location; wherein, the probability distribution corresponding to that spatiotemporal location is used to reflect the relative importance of each channel within the feature representation of the second input feature at that spatiotemporal location; the second input feature is either the third intermediate feature or the fourth intermediate feature;

[0030] The information entropy values ​​corresponding to the second input feature at each spatiotemporal location are combined to obtain the feature of the second input feature compressed position by position along the channel dimension.

[0031] In some embodiments, the encoder-decoder architecture network processes multimedia data; the multimedia data includes at least one of image data and audio data.

[0032] The global features are used to describe the high-level semantic information and overall structural context of the multimedia data; the local features are used to describe the detailed texture and fine structure of the multimedia data.

[0033] According to another aspect of this disclosure, an encoder-decoder architecture network is provided, the network comprising N cascaded decoding network layers; each comprising an upsampling unit and a feature fusion module;

[0034] The upsampling unit of the decoding network layer is used to upsample the features input to the upsampling unit, and the output features of the upsampling unit are used as the input of the feature fusion module.

[0035] The input to the feature fusion module of the decoding network layer also includes: features generated by the network during the encoding stage that have the same spatial resolution as the decoding network layer;

[0036] The feature fusion module of any decoding network layer uses the adaptive feature fusion method based on entropy difference gating, as described above, to generate the fused features of this decoding network layer, which are then used as the output of this decoding network layer.

[0037] According to another aspect of this disclosure, an adaptive feature fusion apparatus based on entropy difference gating is provided, the apparatus being applied to the decoding stage of an encoder-decoder architecture network, the apparatus comprising:

[0038] The channel dimension processing module is configured to construct channel dimension feature pairs based on global and local features in the current network layer of the decoding stage; wherein, the channel dimension feature pair includes a first feature and a second feature; the first feature is obtained by channel-by-channel compression along the spatiotemporal dimension based on the global feature; the second feature is obtained by channel-by-channel compression along the spatiotemporal dimension based on the local feature; the global feature is the feature output from the current network layer after upsampling; the local feature is the feature with the same spatial resolution as the current network layer in the encoding stage of the encoder-decoder architecture network;

[0039] The spatiotemporal dimension processing module is configured to construct spatiotemporal dimension feature pairs based on the global features and local features; wherein, the spatiotemporal dimension feature pairs include a third feature and a fourth feature; the third feature is obtained by compressing the global features position by position along the channel dimension; the fourth feature is obtained by compressing the local features position by position along the channel dimension.

[0040] The deterministic difference acquisition module is configured to determine a first entropy difference based on the channel dimension feature pair and a second entropy difference based on the spatiotemporal dimension feature pair; wherein the first entropy difference and the second entropy difference are both determined based on the same feature comparison direction; the first entropy difference indicates the relative difference in deterministic feature representation between the global features and local features in the channel dimension; the second entropy difference is used to indicate the relative difference in deterministic feature representation between the global features and local features in the spatiotemporal dimension.

[0041] The gating weight determination module is configured to perform gating mapping processing on the first entropy difference and the second entropy difference to generate local gating weights and global gating weights.

[0042] The fusion module is configured to use the local gate weights and global gate weights to perform gated fusion processing on the global features and local features in the current network layer to generate the fused features of the current network layer.

[0043] In some embodiments, the feature comparison direction includes either a global-to-local deterministic comparison direction or a local-to-global deterministic comparison direction; the deterministic difference acquisition module specifically includes:

[0044] If the feature comparison direction is a deterministic comparison direction of the global relative to the local, then the first entropy difference is determined by the entropy increment of the first feature compared to the second feature, and the second entropy difference is determined by the entropy increment of the third feature compared to the fourth feature.

[0045] If the feature comparison direction is a deterministic comparison direction of local relative to global, then the first entropy difference is determined by the entropy increment of the second feature compared to the first feature, and the second entropy difference is determined by the entropy increment of the fourth feature compared to the third feature.

[0046] In some embodiments, the gating weight determination module specifically includes:

[0047] A first gating weight is generated based on the first normalized response of the first entropy difference and the second entropy difference; the first normalized response is obtained by processing the first entropy difference and the second entropy difference respectively through a preset monotonic mapping function.

[0048] A second gating weight is generated based on the second normalized response of the first entropy difference and the second entropy difference; the second normalized response is the element-wise complementary inverse result of the first normalized response.

[0049] Based on the output monotonicity of the monotonic mapping function and the feature comparison direction, a local gate weight is determined from the first gate weight and the second gate weight, and the remaining gate weight among the first gate weight and the second gate weight is determined as the global gate weight.

[0050] In some embodiments, the current network layer includes a first convolutional module and a second convolutional module that have been pre-trained jointly;

[0051] The first convolutional module includes a first convolutional layer and a third convolutional layer that are independent of each other; wherein, the first convolutional layer is used to perform convolution processing on the global features to output a first intermediate feature; the first intermediate feature is obtained by compressing the first feature along the spatiotemporal dimension channel by channel; the third convolutional layer is used to perform convolution processing on the global features to output a third intermediate feature; the third intermediate feature is obtained by compressing the third feature along the channel dimension position by position;

[0052] The second convolutional module includes a second convolutional layer and a fourth convolutional layer that are independent of each other; wherein, the second convolutional layer is used to perform convolution processing on the local features to output a second intermediate feature; the second intermediate feature is obtained by compressing the second feature along the spatiotemporal dimension channel by channel; the fourth convolutional layer is used to perform convolution processing on the local features to output a fourth intermediate feature; the fourth intermediate feature is obtained by compressing the fourth feature along the channel dimension position by position.

[0053] In some embodiments, the first convolutional module further includes a fifth convolutional layer, and the second convolutional module further includes a sixth convolutional layer;

[0054] The parameters of the fifth convolutional layer are independent of the first and third convolutional layers, and are used to perform convolution processing on the global features to output a fifth intermediate feature; the parameters of the sixth convolutional layer are independent of the second and fourth convolutional layers, and are used to perform convolution processing on the local features to output a sixth intermediate feature; the fusion module specifically includes:

[0055] The fifth intermediate feature is weighted using the global gating weight, and the sixth intermediate feature is weighted using the local gating weight. The weighted fifth intermediate feature and the sixth intermediate feature are then fused to generate the fused feature of the current network layer.

[0056] In some embodiments, the channel-by-channel compression along the spatiotemporal dimension includes:

[0057] For each channel of the first input feature, the information entropy value corresponding to that channel is determined based on the probability distribution corresponding to that channel; wherein, the probability distribution corresponding to that channel is used to reflect the relative importance of the first input feature at each spatiotemporal position in the feature representation of that channel; the first input feature is either the first intermediate feature or the second intermediate feature;

[0058] The information entropy values ​​of the first input feature in each channel are combined to obtain the channel-by-channel compressed features along the spatiotemporal dimension of the first input feature.

[0059] In some embodiments, the position-by-position compression along the channel dimension includes:

[0060] For each spatiotemporal location of the second input feature, the information entropy value corresponding to that spatiotemporal location is determined based on the probability distribution corresponding to that spatiotemporal location; wherein, the probability distribution corresponding to that spatiotemporal location is used to reflect the relative importance of each channel within the feature representation of the second input feature at that spatiotemporal location; the second input feature is either the third intermediate feature or the fourth intermediate feature;

[0061] The information entropy values ​​corresponding to the second input feature at each spatiotemporal location are combined to obtain the feature of the second input feature compressed position by position along the channel dimension.

[0062] In some embodiments, the encoder-decoder architecture network processes multimedia data; the multimedia data includes at least one of image data and audio data.

[0063] The global features are used to describe the high-level semantic information and overall structural context of the multimedia data; the local features are used to describe the detailed texture and fine structure of the multimedia data.

[0064] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to perform the steps.

[0065] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0066] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0067] In this embodiment, for any network layer in the encoder-decoder architecture network decoding process, the entropy difference between the global features output by the upsampled network layer and the local features passed in by the skip connections is calculated. This quantifies the deterministic difference in feature representation in the spatiotemporal and channel dimensions. Then, using the quantized deterministic difference as a driving signal, local gate control weights corresponding to the local features and global gate control weights corresponding to the global features are adaptively generated. The local gate control weights and global gate control weights are then used to perform weighted modulation and fusion of the corresponding local and global features. This achieves differentiated feature recalibration under joint channel-spatiotemporal gating, thereby effectively suppressing the interference of low-deterministic features on the fusion result while preserving high-confidence semantic structure and fine spatial details, thus improving the accuracy and robustness of multi-scale feature fusion.

[0068] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0069] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0070] Figure 1A This application exemplarily illustrates a flowchart of an adaptive feature fusion method based on entropy difference gating;

[0071] Figure 1B This application exemplifies an overall flowchart for constructing spatiotemporal dimension feature pairs and channel dimension feature pairs based on global and local features;

[0072] Figure 2 This application exemplarily illustrates a flowchart of determining entropy differences and performing gating mapping processing;

[0073] Figure 3 This application exemplarily illustrates a flowchart of steps for gating and mapping the first entropy difference and the second entropy difference;

[0074] Figure 4A This application exemplarily illustrates a schematic diagram of the structure of the current network layer in the decoding stage;

[0075] Figure 4B This application provides an exemplary schematic diagram illustrating the structure of the current network layer in another decoding stage;

[0076] Figure 5A This application exemplarily illustrates a schematic diagram of a feature fusion module described in this embodiment;

[0077] Figure 5B This application exemplarily illustrates a feature extraction module;

[0078] Figure 5C This application exemplarily illustrates a gate mapping process of a gate generation module;

[0079] Figure 6 This application exemplarily illustrates a virtual device block diagram of an adaptive feature fusion device based on entropy difference gating;

[0080] Figure 7 This application exemplarily illustrates a block diagram of an adaptive feature fusion apparatus based on entropy difference gating. Detailed Implementation

[0081] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0082] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0083] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0084] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0085] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0086] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0087] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0088] Encoder-decoder architecture networks are widely used in various intensive prediction tasks, including image processing (such as medical image segmentation and remote sensing image processing) and audio processing (such as speech separation). This architecture processes information in a bottom-up and then top-down manner, similar to the information processing method in biological vision systems. It includes a network hierarchy for the encoding stage, which abstracts low-level details into high-level semantic representations, and a network hierarchy for the decoding stage, which uses skip connections to fuse preserved spatial details with semantic context upsampled from the network hierarchy.

[0089] Taking the encoder-decoder architecture network U-Net as an example, the U-Net network in related technologies includes an encoder, a decoder, and skip connections. The encoder is used to extract features from the input image and understand its content. It consists of a series of repeating encoding network layers. Each layer includes a convolutional layer and a max-pooling layer. The convolutional layer learns image features from the image, while the max-pooling layer performs downsampling, halving the size of the feature map and doubling the number of feature channels. As the number of feature extraction layers increases, the spatial size of the feature map gradually decreases while the semantic level of the features increases, transforming pixel-level image details into high-dimensional abstract semantic features.

[0090] The decoder is used to restore the feature maps extracted by the encoder to the size of the original image. It consists of a series of repeated decoding network layers. Each network layer includes an upsampling convolution, which doubles the size of the feature map and halves the number of feature channels. The output of the upsampling convolution (i.e., global features) is fused with the feature maps (i.e., local features) from the skip connections.

[0091] The feature map from the skip connection is the feature map output by the corresponding encoding network layer in the encoder that has the same spatial resolution as the current decoding layer. This feature map retains the fine-grained spatial structure and texture information extracted by the encoder at this scale. During the fusion process, it serves as a local feature that supplements the high-resolution details and complements the global features output by the current layer of the decoder, which are rich in high-level semantics but lack spatial details.

[0092] In this network architecture, the effectiveness of the decoding stage depends on the ability of each network layer used to implement decoding to integrate multi-scale information. In related technologies, each network layer used to implement decoding is typically configured with a fusion module to achieve the fusion of multi-scale feature information of its own network layer. This fusion module usually adopts a simple fusion method or an attention-based fusion method.

[0093] Simple fusion methods typically rely on element-wise addition or feature concatenation. While these methods are computationally efficient, they fail to differentiate the importance of features at different scales, indiscriminately processing features from different semantic levels. They lack adaptability to suppress noise or emphasize task-relevant regions, making the integrated output features susceptible to interference from low-quality redundant information, thus weakening feature representation capabilities. Furthermore, they are prone to producing coarse segmentation edges or missegmentation when local features contain significant background noise or artifacts.

[0094] Fusion methods based on attention mechanisms can be divided into selective attention methods and cross-attention methods. Selective attention methods derive attention weights from global features and use these weights to perform gated weighting on local features; while cross-attention methods model the correlation between global and local features through query-key-value projection, capturing richer interactions, but this method has a high computational cost for fusion.

[0095] From a neuroscience perspective, the encoder-decoder structure is similar to the feedforward and feedback pathways in visual and auditory systems. Predictive coding theory posits that the core function of feedback connections is to predict low-level representations and calculate prediction errors. However, the two types of attention-based fusion methods mentioned above typically only use global features as conditional signals to unidirectionally modulate local features. The global features themselves remain fixed throughout the fusion process and are not adaptively adjusted. Moreover, the attention weights are derived only from the original global features themselves or pairwise correlations, failing to clearly reflect the differences between cross-scale feature flows. This ignores the feedback correction effect of local features on global features, resulting in the inability of global features to self-correct based on the actual quality of local information when the local features input from skip connections contain noise or semantic bias. Consequently, this limits the overall quality and robustness of multi-scale feature fusion.

[0096] Therefore, how to balance local detail fidelity and global semantic consistency during the decoding stage, thereby improving the accuracy of multi-scale feature fusion and enhancing the overall feature representation capability of the network, is a technical problem that urgently needs to be solved.

[0097] Based on this, this disclosure provides an adaptive feature fusion method based on entropy difference gating. For any network layer in the decoding process, this method quantifies the deterministic difference in feature representation between the global features output from the upsampled network layer and the local features input from the skip connections by calculating the information entropy difference. Further, based on the quantified deterministic difference, local gating weights and global gating weights corresponding to the local and global features are adaptively generated, respectively. These local and global gating weights are then used to perform weighted modulation and fusion of the corresponding local and global features. This embodiment, by explicitly modeling the deterministic differences between cross-scale features, drives the generation of gating weights, effectively suppressing the interference of low-deterministic features on the fusion result, and improving the accuracy and robustness of multi-scale feature fusion.

[0098] The above method is applied to the decoding stage of the encoder-decoder network. For any network layer used to implement decoding, after obtaining the global features of the upsampled output and the local features of the skip connections at the network layer, the fusion module configured by the relevant technology is replaced with the adaptive feature fusion module based on entropy difference gating provided in this embodiment, so as to realize the fusion of the global features and the local features according to the adaptive feature fusion method based on entropy difference gating provided in this embodiment.

[0099] This method can be applied to encoder-decoder architectures that include encoders, decoders, and cross-scale skip connections, such as U-Net, V-Net, TransUNet, and Swin-UNet, but is not limited to the specific network variants mentioned above. It can be understood that for network structures that introduce multi-scale local features through skip connections in the decoding stage and fuse them with upsampled global features, the adaptive feature fusion method described in this embodiment can be used to replace the original fusion module to achieve adaptive gating modulation based on information entropy differences.

[0100] The adaptive feature fusion method based on entropy difference gating provided in this embodiment can be executed by any computer device with data processing and computing capabilities, such as various terminals or servers. The server can include an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services. The terminal can be a smartphone, tablet computer, laptop computer, or desktop computer, but is not limited to these.

[0101] This method can be applied to multimedia signal analysis scenarios such as computer vision and audio processing, including lesion segmentation in medical images, feature recognition in remote sensing images, speech enhancement, and sound source separation. The encoder-decoder architecture network uses multimedia data from the application scenario as its processing object; the multimedia data includes at least one of image data and audio data. The aforementioned global features describe the high-level semantic information and overall structural context of the multimedia data. These features originate from the deep abstraction of the encoder and are recovered through upsampling by the decoder, carrying the content understanding of the highly compressed and abstracted multimedia data to provide overall content support for the decoding and reconstruction of the multimedia data. The aforementioned local features describe the detailed texture and fine structure of the multimedia data, such as edge contours, surface textures, or transient waveform details, to supplement high-frequency detail information for the decoding and reconstruction of the multimedia data, correct signal ambiguity during the upsampling process, and improve reconstruction fidelity.

[0102] See Figure 1A The flowchart shown is an example of an adaptive feature fusion method with entropy difference gating, which may include at least the following steps:

[0103] S110, in the current network layer of the decoding stage, channel-dimensional feature pairs are constructed based on global features and local features; wherein, the channel-dimensional feature pair includes a first feature and a second feature; the first feature is obtained by compressing the global features channel by channel along the spatiotemporal dimension; the second feature is obtained by compressing the local features channel by channel along the spatiotemporal dimension;

[0104] S120, Based on the global features and local features, construct a spatiotemporal dimension feature pair; wherein, the spatiotemporal dimension feature pair includes a third feature and a fourth feature; the third feature is obtained by compressing the global features position by position along the channel dimension; the fourth feature is obtained by compressing the local features position by position along the channel dimension;

[0105] S130, determine a first entropy difference based on the channel dimension feature pair, and determine a second entropy difference based on the spatiotemporal dimension feature pair; wherein, the first entropy difference and the second entropy difference are both determined based on the same feature comparison direction; the first entropy difference indicates the relative difference in the determinacy of feature representation between the global feature and the local feature in the channel dimension; the second entropy difference is used to indicate the relative difference in the determinacy of feature representation between the global feature and the local feature in the spatiotemporal dimension.

[0106] S140, perform gating mapping processing on the first entropy difference and the second entropy difference to generate local gating weights and global gating weights;

[0107] S150, using the local gate weights and global gate weights, perform gated fusion processing on the global features and local features in the current network layer to generate the fused features of the current network layer.

[0108] This embodiment uses the relative differences in the determinism of global and local features in the spatiotemporal and channel dimensions as a basis to distinguish the importance of global and local features in the multi-scale feature fusion process. Thus, it performs gated weighted fusion by weighting the corresponding features with adaptively generated local gate weights and global gate weights, thereby improving the feature fusion quality and feature representation capability.

[0109] To enable those skilled in the art to understand and implement this disclosure, the specific implementation process of each of the above steps will be briefly described below.

[0110] In this embodiment, the global features are the features output from the current network layer after upsampling, and the local features are the features in the encoding stage of the encoder-decoder architecture network that have the same spatial resolution as the current network layer. The global and local features have the same spatial resolution and, after channel-dimensional alignment, the same number of channels.

[0111] Here, spatial resolution represents the number of discrete sampling points or sequence length of the feature map within the inherent topological structure of the data. Taking an image as an example, spatial resolution is the width W and height H of the feature map; for example, 64×64 means that the feature map has 64 pixel-level sampling points in both the horizontal and vertical directions. Taking audio as an example, spatial resolution is the time step T of the feature sequence; for example, 1024 means that the feature contains 1024 frame-level sampling points on the time axis.

[0112] The number of channels represents the dimension of the feature vector carried at each spatial sampling point, that is, the number of independent semantic attributes or filters learned by the network. Taking images as an example, the number of channels corresponds to the abstract expression of different visual features. For example, RGB three channels represent the primary color, while 64 channels may represent mid-level visual patterns such as encoded edges, textures, and color blocks. Taking audio as an example, the number of channels corresponds to the abstract expression of different acoustic features. For example, 1 channel represents the original mono waveform, while 128 channels represent acoustic attributes such as encoded fundamental frequency, formants, and Mel-frequency cepstral coefficients.

[0113] The current network layer in the decoding stage includes an upsampling convolutional layer, which is used to upsample the feature map input to the current network layer via the adjacent previous network layer. This means restoring the size of the input feature map to the spatial resolution of the current network layer (usually doubling the size) and adjusting the number of channels to match the dimension of the current network layer (usually halving the number of channels). The output of the upsampling is used as the global feature.

[0114] Local features are extracted from the encoding stage and passed to the decoding stage via skip connections. These local features are feature maps from the encoding stage that share the same spatial resolution as the current layer in the decoding stage, and are directly passed to the corresponding layer in the decoder after being copied. For example, during downsampling in the encoding stage, after the i-th network layer's convolutional processing but before the downsampling convolution, a feature map Ei is generated. This feature map is then passed to the network layer in the decoding stage that has the same spatial resolution as the i-th network layer, serving as a local feature for that network layer to fuse with global features.

[0115] Regarding the global and local features, when the processing object is image data, the global features can be tensors output by the upsampling path that describe the high-level semantic features of the image data, and the local features are feature maps of the same resolution at the encoding stage that are passed through skip connections and describe the spatial detail features of the image data (such as edge contours, texture structures, and other pixel-level information).

[0116] When the object being processed is audio data, the global feature can be a time-frequency representation of the acoustic structure features of the audio data (such as fundamental envelope, formant distribution, and other long-term acoustic modes) output by the upsampling path, and the local feature is a time-frequency map of the same resolution at the encoding stage, which is transmitted through skip connections and describes the time-frequency details of the audio data (such as transient impulses, high-frequency harmonics, and other short-term fine structures).

[0117] Steps S110 and S120 above are used to construct entropy difference feature pairs on the determinism of global and local feature representations in each dimension, based on global and local features, from the spatiotemporal dimension and the channel dimension. This reflects the confidence distribution deviation and complementarity strength of global semantics and local details in different feature subspaces, providing an entropy-based differential weighting basis for subsequent adaptive feature fusion. This enables the network to dynamically suppress noise interference in high-entropy (low-determinism) regions and enhance the effective representation transmission in low-entropy (high-determinism) regions.

[0118] In deep learning, feature maps are typically represented as... A four-dimensional tensor, in which spatiotemporal location and channel are two orthogonal dimensions describing the tensor.

[0119] Spatiotemporal location represents a specific combination of time frames and spatial coordinates in a feature map; for example, for an image, since a single image does not contain information about temporal evolution, its temporal dimension degenerates to a constant, and spatiotemporal location represents the pixel spatial coordinates on a two-dimensional spatial plane. For audio, after the audio signal is converted into a two-dimensional time-frequency representation by short-time Fourier transform or Mel filter bank, the spatial dimension is replaced by the frequency axis, and the spatiotemporal position represents the combination of the time frame index and the frequency sub-band on the time-frequency diagram. .

[0120] The channel represents the third dimension C of the feature map. Each channel is a feature response map of the same size as the original input. Each channel essentially corresponds to a specific convolutional kernel / detector learned by the network. Different channels are responsible for encoding different high-level semantic concepts.

[0121] Based on this, channel dimension features describe the information complexity difference of spatiotemporal activation patterns of global and local features in each channel. Therefore, it is necessary to compress or aggregate the spatiotemporal dimension using the channels of the feature map as semantic units, and determine the deterministic information of feature representation in each channel by calculating the probability distribution entropy value of the spatiotemporal response in each channel, thereby quantitatively evaluating the information reliability of each channel in distinguishing between global structure and local details.

[0122] Combination Figure 1B An exemplary flowchart illustrating the overall process of constructing spatiotemporal dimension feature pairs and channel dimension feature pairs based on global and local features shows that, in the channel dimension, a first feature Gn is generated based on global features, and a second feature Ln is generated based on local features. The first feature Gn and the second feature Ln constitute the channel dimension feature pair (Gn, Ln).

[0123] Similarly, spatiotemporal dimension features describe the difference in information complexity between global and local features across channel response distributions at various spatiotemporal locations. Therefore, channel dimensions need to be compressed or aggregated using each spatiotemporal location of the feature map as a perception unit. The deterministic information of feature representation at each spatiotemporal location is determined by calculating the probability distribution entropy value of the cross-channel response at each spatiotemporal location, so as to measure the effective information density of the spatiotemporal location in the global context and local texture.

[0124] refer to Figure 1B As shown, in the spatiotemporal dimension, a third feature Gp is ​​generated based on global features, and a fourth feature Lp is generated based on local features. The third feature Gp and the fourth feature Lp constitute a spatiotemporal dimension feature pair (Gp, Lp).

[0125] For step 130, channel-dimensional feature pairs and spatiotemporal feature pairs are further utilized to analyze the degree of information certainty of global features and local features in the two complementary dimensions of channel and spatiotemporal location. The abstract entropy value is transformed into a numerical signal with clear positive and negative meaning that can directly guide the allocation of fusion weights. This provides a calculable basis for subsequent gating fusion, enabling the network to automatically determine whether to trust global features or local features more in each channel and at each location.

[0126] This step uses channel-dimensional feature pairs as a basis. Based on the spatiotemporal response entropy distribution of each channel included by the first and second features, it evaluates the feature representation determinism of global and local features in each channel to indicate whether the channel should trust global or local features more under the current task, thereby achieving channel-level adaptive selection.

[0127] In addition, based on the spatiotemporal dimension feature pairs, and the cross-channel response entropy distribution at each spatiotemporal location included by the third and fourth features, the feature representation determinism of global features and local features at each spatiotemporal location is evaluated to indicate whether global features or local features should be trusted more under the current task, thus achieving spatiotemporal location-level adaptive positioning.

[0128] Since entropy is only a scalar measure and does not have inherent directionality, if the entropy difference between the channel dimension and the spatiotemporal dimension is calculated using opposite comparison benchmarks, it will lead to a contradiction between the optimization objectives of the channel dimension and the spatiotemporal dimension. Therefore, this embodiment needs to clarify that the first entropy difference and the second entropy difference are both determined based on the same feature comparison direction.

[0129] The feature comparison direction should be understood as the ordered relationship between the reference benchmark and the compared object when calculating the difference between the first and second entropy. In other words, the feature comparison direction defines which entropy value is used as the minuend and which as the subtrahend when calculating the entropy difference, so that the difference values ​​of the two dimensions can be jointly processed and gated under the same set of numerical rules.

[0130] The feature comparison direction can include either a deterministic comparison direction of global relative to local, or a deterministic comparison direction of local relative to global. Based on this, see [link to relevant documentation]. Figure 2 The exemplary flowchart illustrates a process for determining entropy differences and performing gated mapping. Specifically, regarding the process of determining the first entropy difference based on the channel dimension features and the second entropy difference based on the spatiotemporal dimension features, the entropy difference is determined based on different feature comparison directions in the following ways:

[0131] S210, if the feature comparison direction is a deterministic comparison direction of the global relative to the local, then the first entropy difference is determined by the entropy increment of the first feature compared to the second feature, and the second entropy difference is determined by the entropy increment of the third feature compared to the fourth feature.

[0132] The global-to-local deterministic comparison direction represents the measurement of the deterministic gain of global features relative to local features, using local features as a reference benchmark. This feature comparison direction is more suitable for scenarios where local details are considered more reliable in the prior art and global information is needed to supplement or verify them, such as fine-grained recognition tasks where local texture is fundamental and global structure is used for disambiguation.

[0133] In this comparison direction, if the entropy difference is negative, it means that the global features are more certain and the information distribution is more concentrated in this dimension. Gated fusion in this dimension should be more inclined to trust the global features. Conversely, if the entropy difference is positive, it means that the local features are more certain and the information distribution is more concentrated in this dimension. In this case, gated fusion in this dimension should be more inclined to trust the local features.

[0134] Taking the aforementioned channel dimension feature pair (Gn, Ln) and spatiotemporal dimension feature pair (Gp, Lp) as examples, the first entropy difference Δn = Gn - Ln, and the second entropy difference Δp = Gp - Lp.

[0135] S220, if the feature comparison direction is a deterministic comparison direction of local relative to global, then the first entropy difference is determined by the entropy increment of the second feature compared to the first feature, and the second entropy difference is determined by the entropy increment of the fourth feature compared to the third feature.

[0136] The deterministic comparison direction between local and global features uses global features as a reference to measure the deterministic gain of local features relative to global features. This feature comparison direction is more suitable for scenario tasks where the global context is considered more reliable in the prior art, but the local context may be affected by noise, such as remote sensing or medical images where the global structure is stable but the local context is susceptible to artifacts.

[0137] In this comparison direction, if the entropy difference is negative, it means that the local features are more certain and the information distribution is more concentrated in this dimension, and the gating fusion in this dimension should be more inclined to trust the local features; conversely, if the entropy difference is positive, it means that the global features are more certain and the information distribution is more concentrated in this dimension, and the gating fusion in this dimension should be more inclined to trust the global features.

[0138] Taking the aforementioned channel dimension feature pair (Gn, Ln) and spatiotemporal dimension feature pair (Gp, Lp) as examples, in this comparison direction, the first entropy difference Δn = Ln - Gn, and the second entropy difference Δp = Lp - Gp.

[0139] Regarding step S140, the local gate control weight represents the confidence coefficient of the local feature to be assigned to each channel and each spatiotemporal location in the current network layer. Correspondingly, the global gate control weight represents the confidence coefficient of the global feature to be assigned to the global feature in the same dimension. Together, they constitute an adaptive weighted modulation of the global and local features.

[0140] After obtaining the first entropy difference in the channel dimension and the second entropy difference in the spatiotemporal dimension through the aforementioned steps, since the entropy difference reflects a deterministic relative magnitude, its numerical range and symbolic semantics need to be transformed into a normalized weight tensor with a clear trust orientation through gating mapping. This gating mapping process includes at least: compressing the first and second entropy differences into a valid weight range; and correctly assigning the mapping results to local or global features according to the selected feature comparison direction.

[0141] Based on this, for the process of gating and mapping the first entropy difference and the second entropy difference, see [link to documentation]. Figure 3 An exemplary flowchart illustrates the steps of gating and mapping the first entropy difference and the second entropy difference, which may include the following implementation methods:

[0142] S310, Based on the first normalized response of the first entropy difference and the second entropy difference, generate the first gating weight;

[0143] The first normalized response includes the normalized result of the first entropy difference and the normalized result of the second entropy difference, which is obtained by transforming the first entropy difference and the second entropy difference through a preset monotonic mapping function. This monotonic mapping function can be a monotonically increasing or monotonically decreasing function, used to map the first entropy difference and the second entropy difference to the interval [0,1].

[0144] For example, in this embodiment, the sigmoid function is selected as the preset monotonic mapping function. The first entropy difference Δn is normalized to obtain sigmoid(Δn), and the second entropy difference Δp is normalized to obtain sigmoid(Δp). Thus, sigmoid(Δn) and sigmoid(Δp) constitute the first normalized response.

[0145] S320, a second gating weight is generated based on the second normalized response of the first entropy difference and the second entropy difference; the second normalized response is the element-wise complementary inverse result of the first normalized response;

[0146] Element-by-element complementary negation means performing the operation of subtracting the value of each element from 1 for each element in the first normalized response, so that the value of the complementary negation result is still in the interval [0,1] and the sum of the result and the original element is 1.

[0147] Based on this, the second normalized response includes the complementary value of the normalized result of the first entropy difference, and the complementary value of the normalized result of the second entropy difference. Taking the above-mentioned sigmoid(△n) and sigmoid(△p) as the first normalized response, the second normalized response is composed of the following complementary values: 1-sigmoid(△n) and 1-sigmoid(△p).

[0148] It should be understood that since both the first entropy difference and the second entropy difference are scalar sequences calculated along the feature dimension, the first gating weight and the second gating weight mentioned above are also weight vectors.

[0149] The process of generating the first / second gating weights described above can be implemented through broadcast multiplication along the complementary dimension. That is, the normalized results of the first entropy difference and the second entropy difference in the first normalized response are multiplied by broadcast multiplication to generate the first gating weight. Similarly, the complementary values ​​of the normalized results of the first entropy difference and the second entropy difference in the second normalized response are multiplied by broadcast multiplication to generate the second gating weight.

[0150] Taking the first normalized response formed by the above sigmoid(△n) and sigmoid(△p) as an example, based on broadcast multiplication... The process of generating the first gating weight KL can be represented as:

[0151] KL = sigmoid(△n) sigmoid(△p);

[0152] In this first gating weight KL, the (c,j)th element is equal to the product of the gating value of the cth (1≤c≤C)th channel and the gating value of the jth (1≤j≤P)th spatiotemporal position; C represents the number of channels in sigmoid(△n), and P represents the number of spatiotemporal positions in sigmoid(△p).

[0153] For example, suppose C = 3 channels and P = 4 spatiotemporal locations;

[0154] sigmoid(△n) = [0.9, 0.2, 0.7] (shape: 3×1)

[0155] sigmoid(△p)=[0.8,0.1,0.6,0.5](shape: 1×4)

[0156] implement Broadcast multiplication (outer product)

[0157]

[0158]

[0159] By using the broadcast multiplication calculation method for gating mapping, the importance of channels and spatiotemporal positions are coupled, thereby accurately suppressing the characteristic responses of noise positions on important channels and all spatial positions on noise channels. Compared with the addition mapping method where the importance of channels and spatiotemporal positions are independently superimposed, this method can avoid the compensation effect of high-importance channels on low-quality spatiotemporal positions, as well as the masking effect of high-importance spatiotemporal positions on low-quality channels, thus achieving joint selective filtering of channel dimension and spatiotemporal dimension.

[0160] S330, based on the output monotonicity of the monotonic mapping function and the feature comparison direction, determine the local gate weight from the first gate weight and the second gate weight, and determine the remaining gate weight from the first gate weight and the second gate weight as the global gate weight.

[0161] The output monotonicity of a monotonic mapping function reflects the trend of the output value of the monotonic mapping function changing with the entropy difference of the input, including monotonically increasing or monotonically decreasing.

[0162] When the output monotonicity of the monotonic mapping function is monotonically increasing:

[0163] If the feature comparison direction is a deterministic comparison direction of global relative to local, then the first gating weight is used as the local gating weight and the second gating weight is used as the global gating weight; if the feature comparison direction is a deterministic comparison direction of local relative to global, then the first gating weight is used as the global gating weight and the second gating weight is used as the local gating weight.

[0164] Taking the feature comparison direction as a deterministic comparison direction of global relative to local as an example, the first gating weight is KL=sigmoid(△n). sigmoid(△p) outputs a high value close to 1 when the input parameter is greater than 0, and a low value close to 0 when the input parameter is less than 0. Since △n or △p greater than 0 in this feature comparison direction indicates higher certainty for local features, sigmoid(△n) and sigmoid(△p) reflect the confidence in local features, with the first gating weight KL serving as the local gating weight. Correspondingly, 1-sigmoid(△n) and 1-sigmoid(△p) reflect the confidence in global features, with the second gating weight KG serving as the global gating weight.

[0165] Based on the principle of similarity, when the feature comparison direction is a deterministic comparison direction of local relative to global, KL = sigmoid(Δn) is maintained. In the sigmoid(△p) gating mapping method, sigmoid(△n) and sigmoid(△p) reflect the confidence of global features, and the first gating weight KL serves as the global gating weight; correspondingly, 1-sigmoid(△n) and 1-sigmoid(△p) reflect the confidence of local features, and the second gating weight KG serves as the local gating weight.

[0166] When the output monotonicity of the monotonic mapping function is monotonically decreasing:

[0167] If the feature comparison direction is a deterministic comparison direction of global relative to local, then the first gating weight is used as the global gating weight and the second gating weight is used as the local gating weight; if the feature comparison direction is a deterministic comparison direction of local relative to global, then the first gating weight is used as the local gating weight and the second gating weight is used as the global gating weight.

[0168] Based on the aforementioned steps that have yielded local and global gate weights, step 150 uses these local and global gate weights to perform feature modulation and fusion on the global and local features in the current network layer. Before feature fusion, the local and global gate weights need to be normalized before application.

[0169] The fusion feature generated in this step is a comprehensive feature representation constructed by integrating global and local features. It integrates high-level semantic information and overall structural context of multimedia data, as well as detailed texture and fine structure of multimedia data. Through feature fusion operation, the fusion feature has both high-level semantic understanding and fine detail restoration capabilities, thus forming a more comprehensive and accurate feature expression of multimedia data.

[0170] During the decoding process, the fusion features generated by the current network layer are used as input to the next network layer to continue upsampling and fusion in the next network layer until the current network layer is the last network layer in the decoding stage. The fusion features output by the last network layer are used to generate the processing result of the encoder-decoder architecture network on the multimedia data. The specific process of generating the processing result of the multimedia data based on the fusion features is related to the application scenario and task of the encoder-decoder architecture network.

[0171] For example, in image semantic segmentation, where the object of processing is image data, the fusion features output from the last network layer can be mapped through a convolution classifier head to a category probability map with the same spatial resolution as the input image, thereby achieving accurate boundary and semantic annotation of each target region in the image. In speech enhancement and denoising, where the object of processing is audio data, the fusion features output from the last network layer are used to generate an ideal ratio mask or directly output a clean speech waveform. Alternatively, the speech harmonic structure and transient details carried in the fusion features can be used to separate and recover the target human voice signal from noisy frequencies.

[0172] Local gated weights are used to weight and modulate local features or variant features that have undergone further convolutional processing based on local features, while global gated weights are used to weight and modulate global features or variant features that have undergone further convolutional processing based on global features. Therefore, when generating the fused features for the current network layer, the global gated weights can be used to weight the global features, and the local gated weights can be used to weight the local features. The weighted global features and local features are then fused to generate the fused features for the current network layer.

[0173] Alternatively, to enhance the ability of fused features to express key discriminative information and suppress background noise interference, further convolution processing can be performed based on local and global features to extract semantically aligned and spatially refined local enhancement features and global context features. The local enhancement features are then weighted using the local gate control weights, and the global context features are weighted using the global gate control weights. The weighted local enhancement features and global context features are then fused to generate the fused features of the current network layer, thereby improving the robustness of the model's feature representation and task performance in complex scenarios.

[0174] In the fusion stage after the weighting operation, methods such as element-wise addition and feature concatenation followed by convolutional layer integration can be used to fuse the weighted global features with local features (or weighted local enhanced features with global context features). For example, taking weighted local enhanced features and global context features as an example, elements at the same position in the weighted local enhanced features and weighted global context features can be added together, and the output feature after the operation is used as the fused feature. Alternatively, the weighted local enhanced features and weighted global context features can be concatenated to retain complete information, and then convolution can be used to achieve information interaction and dimensionality reduction, with the dimensionality-reduced output feature serving as the fused feature.

[0175] It is understandable that, in actual implementation, other fusion strategies that can achieve multi-source feature information interaction and aggregation can be flexibly selected according to the current processing object and application scenario, and this disclosure does not limit this.

[0176] Based on the above embodiments, this disclosure provides a joint channel-spatiotemporal adaptive gating fusion method driven by entropy difference. This method constructs feature pairs by orthogonally compressing global and local features along both the channel and spatiotemporal dimensions during the decoding stage. It then quantifies the relative differences in the determinism of feature representations in the two dimensions (i.e., the first and second entropy differences), mapping these dual-dimensional entropy differences to local and global gating weights, thus achieving differentiated weighted fusion of the two feature streams. Under this technical approach, on the one hand, by decoupling compression and independent entropy measurement along the channel and spatiotemporal dimensions, a refined perception of feature uncertainty is achieved, avoiding information loss or misjudgment caused by single-dimensional evaluation. On the other hand, by using the dual-dimensional entropy difference determined by the same feature comparison direction as a unified driving signal to generate gating weights, differentiated weighting of global and local features is achieved. This allows the network to dynamically suppress interference components in low-determinism regions while preserving high-confidence semantic structures and fine spatial details, thereby improving the accuracy and robustness of multi-scale feature fusion and enhancing the expressive power of the fused features.

[0177] In some possible embodiments, in order to decouple channel statistical information from spatial structural information to eliminate semantic interference when comparing features of different dimensions, and to enhance the representational adaptability of global features and local features under their respective evaluation dimensions, this embodiment further convolves the global features and local features, and constructs the channel dimension feature pair or spatiotemporal dimension feature pair based on the result of the further convolution.

[0178] like Figure 4A The exemplary diagram illustrates the structure of a current network layer in the decoding stage. The current network layer includes a pre-trained joint first convolutional module M40 and a second convolutional module M41. The first convolutional module M40 includes a first convolutional layer 401 and a third convolutional layer 403, which are independent of each other. The second convolutional module M41 includes a second convolutional layer 402 and a fourth convolutional layer 404, which are independent of each other.

[0179] The parameter configurations of the first convolutional layer 401 and the third convolutional layer 403 are independent of each other and are determined through the aforementioned pre-joint training. The first convolutional layer 401 is used to perform convolution processing on the global feature G to output a first intermediate feature Z1; the first intermediate feature is compressed channel-wise along the spatiotemporal dimension to obtain the first feature Gn; the third convolutional layer 403 is used to perform convolution processing on the global feature G to output a third intermediate feature Z3; the third intermediate feature is compressed position-wise along the channel dimension to obtain the third feature Gp.

[0180] The parameter configurations of the second convolutional layer 402 and the fourth convolutional layer 404 are independent of each other, and the parameter configurations are determined through the aforementioned pre-joint training. The second convolutional layer 402 is used to perform convolution processing on the local feature L to output a second intermediate feature Z2; the second intermediate feature is compressed channel-wise along the spatiotemporal dimension to obtain the second feature Ln; the fourth convolutional layer 404 is used to perform convolution processing on the local feature L to output a fourth intermediate feature Z4; the fourth intermediate feature is compressed position-wise along the channel dimension to obtain the fourth feature Lp.

[0181] Therefore, the channel dimension feature pair consists of the first feature Gn and the second feature Ln, and the spatiotemporal dimension feature consists of the third feature Gp and the fourth feature Lp.

[0182] Based on the original global and local feature maps, channel response intensity and spatiotemporal activation patterns are highly coupled. If the same set of features is used directly to calculate channel entropy difference and spatiotemporal entropy difference, the two metrics will contaminate each other. This embodiment learns different projection subspaces through independent convolutional kernels, and then, in conjunction with orthogonal compression directions, forces physical isolation between the two types of information. This ensures that the first entropy difference reflects the deterministic difference in channel characteristics, and the second entropy difference reflects the deterministic difference in spatiotemporal structure, thereby improving the accuracy of gating decisions.

[0183] Given that the first entropy difference and the second entropy difference are determined based on the further convolution results of the aforementioned global and local features, when determining the local gate control weights and global gate control weights based on the first entropy difference and the second entropy difference, the gate control weights are actually generated in the evaluation subspace after specific convolution transformation and dimensionality compression. In order to adapt to the gate control weights, this embodiment introduces independent fifth and sixth convolutional layers to project the original global features and the original local features to a refined content feature representation space that is adapted to the gate control weights, so that the weighting operation is performed in a semantically aligned unified space, thereby ensuring that the gate control decision can accurately and effectively guide the generation of fused features.

[0184] See Figure 4B An exemplary schematic diagram of the current network layer structure in another decoding stage is shown. The aforementioned first convolutional module M40 further includes a fifth convolutional layer 405, and the second convolutional module M41 further includes a sixth convolutional layer 406. The parameters of the fifth convolutional layer are configured independently of the first and third convolutional layers, and it is used to perform convolution processing on the global feature G to output a fifth intermediate feature Z5. The parameters of the sixth convolutional layer are configured independently of the second and fourth convolutional layers, and it is used to perform convolution processing on the local feature L to output a sixth intermediate feature Z6.

[0185] In this embodiment, the fifth intermediate feature Z5 is the refined content feature Gd corresponding to the global feature, and the sixth intermediate feature Z6 is the refined content feature Ld corresponding to the global feature. When using the local gate control weight and the global gate control weight to perform gated fusion processing on the global features and local features in the current network layer, the fifth intermediate feature can be weighted using the global gate control weight, and the sixth intermediate feature can be weighted using the local gate control weight. The weighted fifth intermediate feature and the sixth intermediate feature are then fused to generate the fused feature of the current network layer.

[0186] In this embodiment, by setting the fifth and sixth convolutional layers with independent parameters as dedicated content refinement branches, the compressed features (Gn, Ln, Gp, Lp) used to construct entropy difference feature pairs serve only deterministic evaluation, while the refined content features (Gd, Ld) that retain the complete spatial structure and channel details are dedicated to the fusion task. The evaluation-representation dual-path architecture ensures the statistical stability of the gating weight calculation and ensures that the final fused features have sufficient semantic richness and reconstruction capability, thereby improving the feature recovery quality in the decoding stage without increasing the noise of the gating decision.

[0187] In some possible embodiments, the channel-by-channel compression along the spatiotemporal dimension described in any of the foregoing embodiments is essentially the determination of spatiotemporal entropy on each channel of the feature map based on the distribution of activation response intensity of that channel at each spatiotemporal location. The spatiotemporal entropy of any channel reflects the concentration of response or activation determinism of the semantic pattern encoded by that channel in the spatiotemporal domain. The lower the spatiotemporal entropy, the more concentrated the activation energy of that channel is in a few key spatiotemporal locations, that is, the more explicit the response of that channel to a specific local structure and the less redundant background noise.

[0188] Based on this, channel-by-channel compression along the spatiotemporal dimension can be achieved through the following steps:

[0189] For each channel of the first input feature, the information entropy value corresponding to that channel is determined based on the probability distribution corresponding to that channel; wherein, the probability distribution corresponding to that channel is used to reflect the relative importance of the first input feature at each spatiotemporal position in the feature representation of that channel; the first input feature is either the first intermediate feature or the second intermediate feature; the information entropy values ​​of the first input feature corresponding to each channel are combined to obtain the feature of the first input feature compressed along the spatiotemporal dimension channel by channel.

[0190] The probability distribution corresponding to a channel is obtained by normalizing the feature representation of the first input feature corresponding to that channel. That is, the feature representation vector of that channel is regarded as a distribution in the spatiotemporal dimension, and the softmax or other normalized probability function is applied to the feature representation vector to generate the probability distribution corresponding to the channel.

[0191] After obtaining the probability distribution corresponding to this channel Then, the information entropy value corresponding to that channel can be determined using Shannon entropy. That is, for channel c, the information entropy value can be expressed as:

[0192] ;

[0193] q represents the q-th spatiotemporal position within channel c. This represents the probability distribution corresponding to channel c. The probability value at the q-th spatiotemporal location.

[0194] The position-by-position compression along the channel dimension described in any of the foregoing embodiments is essentially about determining the channel entropy at each spatiotemporal position of the feature map based on the distribution of response intensity at each position in each channel dimension. The channel entropy at any spatiotemporal position reflects the specificity or channel selectivity of the semantic expression at that position. The lower the channel entropy, the more it indicates that the activation at that position is contributed by only a few dominant channels, indicating that the position has a clear semantic affiliation rather than a state of multiple semantic mixtures.

[0195] Based on this, position-by-position compression along the channel dimension can be achieved through the following steps:

[0196] For each spatiotemporal location of the second input feature, the information entropy value corresponding to that spatiotemporal location is determined based on the probability distribution corresponding to that location; wherein, the probability distribution corresponding to that spatiotemporal location is used to reflect the relative importance of each channel within the feature representation of the second input feature at that spatiotemporal location; the second input feature is the third intermediate feature or the fourth intermediate feature; the information entropy values ​​of the second input feature at each spatiotemporal location are combined to obtain the feature of the second input feature compressed position by position along the channel dimension.

[0197] Similarly, the probability distribution corresponding to the spatiotemporal location is obtained by normalizing the multi-channel feature representation of the second input feature corresponding to that spatiotemporal location. That is, the multi-channel feature representation vector of that spatiotemporal location is regarded as a distribution in the channel dimension, and softmax or other normalized probability functions are applied to the feature representation vector to generate the probability distribution corresponding to that spatiotemporal location.

[0198] To obtain the probability distribution corresponding to this spatiotemporal location Then, the information entropy value corresponding to the channel can be determined using Shannon entropy. That is, for a spatiotemporal location q, the information entropy value can be expressed as:

[0199] ;

[0200] Represents the probability distribution corresponding to a spatiotemporal location q. The probability value of the c-th channel.

[0201] It is understood that when calculating the information entropy value corresponding to a channel or spatiotemporal location, variance, L2 norm, learnable fully connected layer, etc. can be used to replace Shannon entropy, or other methods can be flexibly selected if the adopted metric can characterize the degree of response concentration or activation determinism of the channel at each spatiotemporal location. This disclosure does not limit this.

[0202] Based on the feature fusion method described above, this embodiment also proposes an encoder-decoder architecture network. This network includes N cascaded decoding network layers; each includes an upsampling unit and a feature fusion module. The upsampling unit of each decoding network layer is used to upsample the features input to it, and the output features of the upsampling unit serve as the input to the feature fusion module. The input to the feature fusion module of each decoding network layer further includes features generated by the network during the encoding stage that have the same spatial resolution as the current decoding network layer.

[0203] Unlike related technologies, the feature fusion module of any decoding network layer in this embodiment uses the adaptive feature fusion method based on entropy difference gating as described in any of the foregoing embodiments to generate the fused features of this decoding network layer as the output of this decoding network layer.

[0204] The feature fusion module described in this embodiment serves as a general fusion component and can be integrated into different encoder-decoder architectures. In medical image segmentation tasks, the standard stitching fusion module in the U-Net and TransAttUNet architectures of related technologies can be replaced with the feature fusion module described in this embodiment, thereby outputting fused features using the feature fusion method provided in this embodiment. In remote sensing image cloud removal tasks, the selective attention fusion in the PMAA model of related technologies can be replaced with the feature fusion module described in this embodiment. In speech separation tasks, the selective attention mechanism in TDANet and TIGER of related technologies can be replaced with the feature fusion module described in this embodiment.

[0205] Unlike attention fusion methods in related technologies that only modulate a single feature stream, the feature fusion module described in this embodiment modulates global and local content features simultaneously through their respective gating maps, achieving a more balanced fusion and effectively improving the expressive power of the fused features.

[0206] To facilitate a better understanding of this disclosure by those skilled in the art, the following description will use the i-th layer of the decoder in the U-Net architecture network as an example.

[0207] This disclosure replaces the fusion modules in each network layer of the decoder of the U-Net architecture network in related technologies with the feature fusion module described in this embodiment. Unlike existing fusion methods, the feature fusion module described in this embodiment dynamically adjusts the contribution of global and local features in the current network layer to the fused features by quantifying the entropy difference between two feature streams.

[0208] Let G∈R^(C×P) represent the global features from the decoder, and L∈R^(C×P') represent the local features from the encoder via skip connections. This feature fusion module adopts a unified framework suitable for both visual and auditory tasks. In computer vision tasks, P and P' correspond to the number of spatial locations (pixels); in audio processing tasks, P and P' represent the number of time steps or time-frequency bins.

[0209] See Figure 5A The schematic diagram of a feature fusion module according to this embodiment is shown as an example. The feature fusion module includes a feature extraction module and a gating generation module.

[0210] The feature extraction module can adopt a three-branch architecture, such as... Figure 5B The diagram illustrates an exemplary feature extraction module, which includes three independent convolutional layers, each followed by a normalization layer and an activation layer. The three independent convolutional layers process the input feature X, and after normalization and activation, output three corresponding intermediate features. , , Intermediate features Channel-wise features are generated through compression along the spatiotemporal dimension. intermediate features Position-wise features are generated by compression along the channel dimension. intermediate features Refined content features used for fusion .

[0211] When compressing and generating channel-wise features along the spatiotemporal dimension, and when compressing and generating position-wise features along the channel dimension, Shannon entropy can be calculated along complementary dimensions to capture the representational determinism of each feature stream. For the process of compressing and generating channel-wise features along the spatiotemporal dimension, for each channel c, the vector... Consider it as a distribution along the spatiotemporal dimension Q, apply softmax along Q to obtain the probability distribution Pn, and then calculate the Shannon entropy: For the process of compressing and generating position-wise features along the channel dimension, for each spatiotemporal location q, the vector is... Treating it as a distribution on the channel, the calculation is similar: Lower entropy indicates that feature activations are concentrated in a few locations (or channels), reflecting higher representational certainty; higher entropy implies a more dispersed and uncertain distribution.

[0212] Based on this, such as Figure 5A As shown, the feature extraction module performs convolution and compression processing on global features G and local features L. After processing global features G, it produces three outputs: ∈R^(C×1) (channel-wise features obtained by compression along the spatiotemporal dimension) ∈R^(1×P) (position-wise features obtained by compression along the channel dimension) and ∈R^(C×P) (refined content features used for fusion); similarly, processing the local feature L produces three outputs: ∈R^(C×1), ∈R^(1×P') and ∈R^(C×P').

[0213] The resulting six output tensors will serve as input to the gating generation module, which will calculate entropy difference measures along the channel and spatiotemporal dimensions to derive the adaptive gating map. Finally, the upsampled global content features... and local content features They are modulated and fused by their respective gating maps to obtain F∈R^(C×P').

[0214] The gating generation module is a decision component that determines the fusion weights by evaluating the differences between the global and local compressed feature pairs generated by the feature extraction module. See also Figure 5C An exemplary diagram illustrates the gating mapping process of a gating generation module, where entropy difference metric captures signed deterministic differences based on entropy scores: Δn = , Δp =φ( )- , where φ(·) represents nearest neighbor interpolation used for resolution alignment. Negative values ​​indicate that global features have lower entropy, thus the feature representation is more deterministic and should be given priority during feature fusion; positive values ​​indicate that local features are more reliable.

[0215] Given Δn and Δp, the gating generation module generates two multiplicative gating mappings:

[0216] = σ(Δn) σ(Δp);

[0217] = (1-σ(Δn)) (1-σ(Δp));

[0218] Where σ(·) represents the sigmoid function. This represents broadcast multiplication along complementary dimensions. Broadcast multiplication is used here to implement the coupled gating mapping design. Normalization ensures that the order of the two gating mappings is determined solely by the entropy difference metric, thus achieving joint channel-spacetime gating.

[0219] The obtained gating mapping , After normalization, KL_norm and KG_norm are generated and applied to refine content features. and The final output fused feature F can be expressed as:

[0220] F = φ( ) KG_norm + KL_norm; This indicates element-wise multiplication.

[0221] This embodiment quantifies the representation determinism of each feature stream using Shannon information entropy, and uses the signed entropy difference between global and local features as a gating signal. It uses the representation determinism reflected by the entropy difference to drive the generation of adaptive gating mapping, and modulates the two feature streams, global and local. It adaptively favors the feature stream with lower entropy along the channel and spatiotemporal dimensions, thereby improving the expressive power of the fused features.

[0222] To verify the effectiveness and universality of the adaptive feature fusion method based on entropy difference gating proposed in this embodiment, the inventors conducted comparative experiments in three task domains: medical image segmentation, remote sensing image cloud removal, and speech separation. The following section, combining specific datasets and the quantization results of baseline models, explains the technical advantages of the feature fusion module described in this embodiment in different application scenarios.

[0223] The feature fusion method proposed in this embodiment achieves performance improvements on multiple tasks. See Table 1 for an example of the performance comparison results of different feature fusion methods on the Synapse dataset using U-Net:

[0224]

[0225] As shown in Table 1, on the Synapse medical image segmentation dataset, using U-Net as the backbone network, the feature fusion module EDG described in this embodiment outperforms all existing fusion methods in both the average Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95).

[0226] DSC is used to measure the degree of overlap between the predicted segmentation region and the ground truth annotation, with a value range of [0, 1] (i.e., 0%–100%), and a higher value is better. The EDG module described in this embodiment achieves 82.96% ± 0.27%, which is +2.98 percentage points higher than the second-best method (stitching, 79.98%).

[0227] HD95 reflects the maximum error of the segmentation boundary, representing the farthest distance at the 95th percentile between the predicted contour and the true contour; a lower value is better. The EDG module described in this embodiment has a diameter of 14.52 ± 0.96 mm, which is 44.1% smaller than the stitching method (25.91 mm), thus improving boundary positioning accuracy.

[0228] In addition, using TransAttUNet as the backbone network, the EDG module described in this embodiment improves the DSC by 1.33 percentage points after replacing the original fusion method of TransAttUNet.

[0229] PSNR (Peak Signal-to-Noise Ratio) is used to measure the difference between the reconstructed image and the ground truth image at the pixel level. The higher the value, the smaller the pixel error and the lower the image distortion.

[0230] SSIM (Structural Similarity Index Measure) is used to measure the similarity between two images in three dimensions: brightness, contrast, and structure. The value range is usually from 0 to 1.

[0231] FID (Frechet Inception Distance) measures the distance between the feature distribution of the generated / reconstructed image and the feature distribution of the real image; a lower value indicates a smaller distance between the two.

[0232] In the task of cloud removal from remote sensing images, the EDG module described in this embodiment, after replacing the selective attention fusion of PMAA, achieves a PSNR of 28.095 dB on the Sen2_MTC_Old dataset, which is +0.718 percentage points higher than the existing PMAA structure. The SSIM is 0.877, and the learned perceptual patch similarity (LPIPS) is 0.291. On the Sen2_MTC_New dataset, the performance of the EDG module replacing PMAA's selective attention fusion is as follows: PSNR = 19.157 dB, SSIM = 0.702, and Fraser inception distance (FID) = 92.315. Compared to the performance of the existing PMAA on this dataset (PSNR = 18.369, SSIM = 0.614, FID = 118.214), PSNR and SSIM are significantly improved, and the FID value decreases, achieving optimization in reconstruction quality, structural fidelity, and perceptual similarity.

[0233] See Table 2 for an example comparison of speech separation performance of different network architectures on the LRS2-2Mix and EchoSet datasets:

[0234]

[0235] As shown in Table 2, in the speech separation task, after integrating the feature fusion module described in this embodiment into TDANet to form TDANet-EDG, the signal-to-interference ratio (SIR) improvement SDRi and scale-invariant SIR improvement SI-SDRi were improved by 0.8 dB on the LRS2-2Mix dataset, and by 2.6 dB and 2.3 dB on the EchoSet dataset, respectively. It is evident that the feature fusion module EDG described in this embodiment effectively enhances the purity and intelligibility of the target speech without introducing amplitude bias.

[0236] After integrating the feature fusion module described in this embodiment into the TIGER architecture to form TIGER-EDG, SI-SDRi reached 16.0 dB on the LRS2-2Mix dataset, which is 0.9 dB higher than the baseline TIGER's 15.1 dB. SI-SDRi reached 15.0 dB on the EchoSet dataset, which is 1.3 dB higher than the baseline TIGER's 15.1 dB.

[0237] Furthermore, the computational overhead introduced by replacing the feature fusion modules in the relevant technical architecture with the feature fusion module of this embodiment is minimal. For example, replacing the feature fusion module described in this embodiment with TDANet only increases the number of parameters by 0.06M and MACs by 0.55G, while replacing the feature fusion module described in this embodiment with TIGER only increases the number of parameters by 0.10M and MACs by 1.76G.

[0238] Corresponding to the aforementioned adaptive feature fusion method based on entropy difference gating, see [link to relevant documentation]. Figure 6 The block diagram of an adaptive feature fusion device based on entropy difference gating is shown as an example. This embodiment also provides an adaptive feature fusion device based on entropy difference gating, which is applied to the decoding stage of an encoder-decoder architecture network. The device includes:

[0239] The channel dimension processing module 610 is configured to construct channel dimension feature pairs based on global and local features in the current network layer of the decoding stage; wherein, the channel dimension feature pair includes a first feature and a second feature; the first feature is obtained by compressing the global feature channel by channel along the spatiotemporal dimension; the second feature is obtained by compressing the local feature channel by channel along the spatiotemporal dimension; the global feature is the feature output by upsampling of the current network layer; the local feature is the feature with the same spatial resolution as the current network layer in the encoding stage of the encoder-decoder architecture network;

[0240] The spatiotemporal dimension processing module 620 is configured to construct a spatiotemporal dimension feature pair based on the global features and local features; wherein, the spatiotemporal dimension feature pair includes a third feature and a fourth feature; the third feature is obtained by compressing the global features position by position along the channel dimension; the fourth feature is obtained by compressing the local features position by position along the channel dimension.

[0241] The deterministic difference acquisition module 630 is configured to determine a first entropy difference based on the channel dimension feature pair and a second entropy difference based on the spatiotemporal dimension feature pair; wherein the first entropy difference and the second entropy difference are both determined based on the same feature comparison direction; the first entropy difference indicates the relative difference in the deterministic representation of the global features and local features in the channel dimension; the second entropy difference is used to indicate the relative difference in the deterministic representation of the global features and local features in the spatiotemporal dimension.

[0242] The gating weight determination module 640 is configured to perform gating mapping processing on the first entropy difference and the second entropy difference to generate local gating weights and global gating weights.

[0243] The fusion module 650 is configured to use the local gate weights and global gate weights to perform gated fusion processing on the global features and local features in the current network layer to generate fused features for the current network layer.

[0244] In some possible embodiments, the feature comparison direction includes either a global-to-local deterministic comparison direction or a local-to-global deterministic comparison direction; the deterministic difference acquisition module specifically includes:

[0245] If the feature comparison direction is a deterministic comparison direction of the global relative to the local, then the first entropy difference is determined by the entropy increment of the first feature compared to the second feature, and the second entropy difference is determined by the entropy increment of the third feature compared to the fourth feature.

[0246] If the feature comparison direction is a deterministic comparison direction of local relative to global, then the first entropy difference is determined by the entropy increment of the second feature compared to the first feature, and the second entropy difference is determined by the entropy increment of the fourth feature compared to the third feature.

[0247] In some possible embodiments, the gating weight determination module specifically includes:

[0248] A first gating weight is generated based on the first normalized response of the first entropy difference and the second entropy difference; the first normalized response is obtained by processing the first entropy difference and the second entropy difference respectively through a preset monotonic mapping function.

[0249] A second gating weight is generated based on the second normalized response of the first entropy difference and the second entropy difference; the second normalized response is the element-wise complementary inverse of the first normalized response;

[0250] Based on the output monotonicity of the monotonic mapping function and the feature comparison direction, a local gate weight is determined from the first gate weight and the second gate weight, and the remaining gate weight among the first gate weight and the second gate weight is determined as the global gate weight.

[0251] In some possible embodiments, the current network layer includes a pre-jointly trained first convolutional module and a second convolutional module;

[0252] The first convolutional module includes a first convolutional layer and a third convolutional layer that are independent of each other;

[0253] Wherein, the first convolutional layer is used to perform convolution processing on the global features to output a first intermediate feature; the first intermediate feature is obtained by compressing the first feature along the spatiotemporal dimension channel by channel; the third convolutional layer is used to perform convolution processing on the global features to output a third intermediate feature; the third intermediate feature is obtained by compressing the third feature along the channel dimension position by position;

[0254] The second convolutional module includes a second convolutional layer and a fourth convolutional layer that are independent of each other;

[0255] The second convolutional layer is used to perform convolution processing on the local features to output a second intermediate feature; the second intermediate feature is obtained by compressing the second feature along the spatiotemporal dimension channel by channel; the fourth convolutional layer is used to perform convolution processing on the local features to output a fourth intermediate feature; the fourth intermediate feature is obtained by compressing the fourth feature along the channel dimension position by position.

[0256] In some possible embodiments, the first convolutional module further includes a fifth convolutional layer, and the second convolutional module further includes a sixth convolutional layer;

[0257] The parameters of the fifth convolutional layer are independent of the first and third convolutional layers, and are used to perform convolution processing on the global features to output a fifth intermediate feature; the parameters of the sixth convolutional layer are independent of the second and fourth convolutional layers, and are used to perform convolution processing on the local features to output a sixth intermediate feature; the fusion module specifically includes:

[0258] The fifth intermediate feature is weighted using the global gating weight, and the sixth intermediate feature is weighted using the local gating weight. The weighted fifth intermediate feature and the sixth intermediate feature are then fused to generate the fused feature of the current network layer.

[0259] In some possible embodiments, the channel-by-channel compression along the spatiotemporal dimension includes:

[0260] For each channel of the first input feature, the information entropy value corresponding to that channel is determined based on the probability distribution corresponding to that channel; wherein, the probability distribution corresponding to that channel is used to reflect the relative importance of the first input feature at each spatiotemporal position in the feature representation of that channel; the first input feature is either the first intermediate feature or the second intermediate feature;

[0261] The information entropy values ​​of the first input feature in each channel are combined to obtain the channel-by-channel compressed features along the spatiotemporal dimension of the first input feature.

[0262] In some possible embodiments, the position-by-position compression along the channel dimension includes:

[0263] For each spatiotemporal location of the second input feature, the information entropy value corresponding to that spatiotemporal location is determined based on the probability distribution corresponding to that spatiotemporal location; wherein, the probability distribution corresponding to that spatiotemporal location is used to reflect the relative importance of each channel within the feature representation of the second input feature at that spatiotemporal location; the second input feature is either the third intermediate feature or the fourth intermediate feature;

[0264] The information entropy values ​​corresponding to the second input feature at each spatiotemporal location are combined to obtain the feature of the second input feature compressed position by position along the channel dimension.

[0265] In some possible embodiments, the encoder-decoder architecture network processes multimedia data; the multimedia data includes at least one of image data and audio data.

[0266] The global features are used to describe the high-level semantic information and overall structural context of the multimedia data; the local features are used to describe the detailed texture and fine structure of the multimedia data.

[0267] The apparatus provided in this disclosure can have functions or include modules that can be used to execute the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0268] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0269] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0270] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0271] Figure 7 This is a block diagram illustrating an apparatus 700 for the aforementioned pipeline network construction method according to an exemplary embodiment. For example, apparatus 700 may be provided as a server or terminal device. (Refer to...) Figure 7 The apparatus 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform the methods described above.

[0272] The device 700 may also include a power supply component 726 configured to perform power management of the device 700, a wired or wireless network interface 750 configured to connect the device 700 to a network, and an input / output interface 758 (I / O interface). The device 700 can operate on an operating system stored in memory 732, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™, or similar.

[0273] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 732 including computer program instructions that can be executed by the processing component 722 of the device 700 to perform the above-described method.

[0274] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0275] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0276] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0277] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0278] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0279] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0280] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0281] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An adaptive feature fusion method based on entropy difference gating, characterized in that, The method is applied to the decoding stage in an encoder-decoder architecture network, and the method includes: In the current network layer of the decoding stage, channel-dimensional feature pairs are constructed based on global and local features; wherein, the channel-dimensional feature pair includes a first feature and a second feature; the first feature is obtained by compressing the global feature channel by channel along the spatiotemporal dimension; the second feature is obtained by compressing the local feature channel by channel along the spatiotemporal dimension; the global feature is the feature output of the current network layer after upsampling; the local feature is the feature with the same spatial resolution as the current network layer in the encoding stage of the encoder-decoder architecture network; Based on the global and local features, a spatiotemporal dimension feature pair is constructed; wherein, the spatiotemporal dimension feature pair includes a third feature and a fourth feature; the third feature is obtained by compressing the global features position by position along the channel dimension; the fourth feature is obtained by compressing the local features position by position along the channel dimension. A first entropy difference is determined based on the channel dimension feature pair, and a second entropy difference is determined based on the spatiotemporal dimension feature pair; wherein, the first entropy difference and the second entropy difference are both determined based on the same feature comparison direction; the first entropy difference indicates the relative difference in the determinacy of feature representation between the global feature and the local feature in the channel dimension; the second entropy difference is used to indicate the relative difference in the determinacy of feature representation between the global feature and the local feature in the spatiotemporal dimension; The first entropy difference and the second entropy difference are subjected to gating mapping to generate local gating weights and global gating weights. Using the local and global gate weights, the global and local features in the current network layer are subjected to gated fusion processing to generate the fused features of the current network layer.

2. The method according to claim 1, characterized in that, The feature comparison direction includes either a global deterministic comparison direction relative to the local or a local deterministic comparison direction relative to the global. The determination of the first entropy difference based on the channel dimension feature pair and the determination of the second entropy difference based on the spatiotemporal dimension feature pair include: If the feature comparison direction is a deterministic comparison direction of the global relative to the local, then the first entropy difference is determined by the entropy increment of the first feature compared to the second feature, and the second entropy difference is determined by the entropy increment of the third feature compared to the fourth feature. If the feature comparison direction is a deterministic comparison direction of local relative to global, then the first entropy difference is determined by the entropy increment of the second feature compared to the first feature, and the second entropy difference is determined by the entropy increment of the fourth feature compared to the third feature.

3. The method according to claim 2, characterized in that, The step of performing gating mapping processing on the first entropy difference and the second entropy difference to generate local gating weights and global gating weights includes: A first gating weight is generated based on the first normalized response of the first entropy difference and the second entropy difference; the first normalized response is obtained by processing the first entropy difference and the second entropy difference respectively through a preset monotonic mapping function. A second gating weight is generated based on the second normalized response of the first entropy difference and the second entropy difference; the second normalized response is the element-wise complementary inverse result of the first normalized response. Based on the output monotonicity of the monotonic mapping function and the feature comparison direction, a local gate weight is determined from the first gate weight and the second gate weight, and the remaining gate weight among the first gate weight and the second gate weight is determined as the global gate weight.

4. The method according to claim 1, characterized in that, The current network layer includes a pre-trained joint first convolutional module and a second convolutional module; The first convolutional module includes a first convolutional layer and a third convolutional layer that are independent of each other; Wherein, the first convolutional layer is used to perform convolution processing on the global features to output a first intermediate feature; the first intermediate feature is obtained by compressing the first feature along the spatiotemporal dimension channel by channel; the third convolutional layer is used to perform convolution processing on the global features to output a third intermediate feature; the third intermediate feature is obtained by compressing the third feature along the channel dimension position by position; The second convolutional module includes a second convolutional layer and a fourth convolutional layer that are independent of each other; The second convolutional layer is used to perform convolution processing on the local features to output a second intermediate feature; the second intermediate feature is obtained by compressing the second feature along the spatiotemporal dimension channel by channel; the fourth convolutional layer is used to perform convolution processing on the local features to output a fourth intermediate feature; the fourth intermediate feature is obtained by compressing the fourth feature along the channel dimension position by position.

5. The method according to claim 4, characterized in that, The first convolutional module further includes a fifth convolutional layer, and the second convolutional module further includes a sixth convolutional layer; The parameters of the fifth convolutional layer are independent of the first and third convolutional layers, and are used to perform convolution processing on the global features to output the fifth intermediate feature; the parameters of the sixth convolutional layer are independent of the second and fourth convolutional layers, and are used to perform convolution processing on the local features to output the sixth intermediate feature. The process of using the local and global gate weights to perform gated fusion processing on global and local features in the current network layer includes: The fifth intermediate feature is weighted using the global gating weight, and the sixth intermediate feature is weighted using the local gating weight. The weighted fifth intermediate feature and the sixth intermediate feature are then fused to generate the fused feature of the current network layer.

6. The method according to claim 4, characterized in that, The channel-by-channel compression along the spatiotemporal dimension includes: For each channel of the first input feature, the information entropy value corresponding to that channel is determined based on the probability distribution corresponding to that channel; wherein, the probability distribution corresponding to that channel is used to reflect the relative importance of the first input feature at each spatiotemporal position in the feature representation of that channel; the first input feature is either the first intermediate feature or the second intermediate feature; The information entropy values ​​of the first input feature in each channel are combined to obtain the channel-by-channel compressed features along the spatiotemporal dimension of the first input feature.

7. The method according to claim 4, characterized in that, The position-by-position compression along the channel dimension includes: For each spatiotemporal location of the second input feature, the information entropy value corresponding to that spatiotemporal location is determined based on the probability distribution corresponding to that spatiotemporal location; wherein, the probability distribution corresponding to that spatiotemporal location is used to reflect the relative importance of each channel within the feature representation of the second input feature at that spatiotemporal location; the second input feature is either the third intermediate feature or the fourth intermediate feature; The information entropy values ​​corresponding to the second input feature at each spatiotemporal location are combined to obtain the feature of the second input feature compressed position by position along the channel dimension.

8. The method according to claim 1, characterized in that, The encoder-decoder architecture network processes multimedia data; the multimedia data includes at least one of image data and audio data. The global features are used to describe the high-level semantic information and overall structural context of the multimedia data; the local features are used to describe the detailed texture and fine structure of the multimedia data.

9. An encoder-decoder architecture network, characterized in that, The network comprises N cascaded decoding network layers; each includes an upsampling unit and a feature fusion module. The upsampling unit of the decoding network layer is used to upsample the features input to the upsampling unit, and the output features of the upsampling unit are used as the input of the feature fusion module. The input to the feature fusion module of the decoding network layer also includes: features generated by the network during the encoding stage that have the same spatial resolution as the decoding network layer; The feature fusion module of any decoding network layer uses the method described in any one of claims 1-8 to generate the fusion feature of the decoding network layer as the output of the decoding network layer.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images