A mirror detection method and system with cross-scale multi-directional feature fusion enhancement

Abstract

The present invention discloses a mirror detection method and system using cross-scale, multi-directional feature fusion enhancement. The method comprises extracting mirror features at M scales from an input image using an encoder, extracting mirror edge features using an edge extraction network, and extracting enhanced features using M layers of feature enhancement blocks (HFFs). The enhanced features are then upsampled and fused at different scales, and then decoded together with the mirror edge features using a decoding head to obtain mirror detection results. The feature enhancement block HFF at any i-th layer consists of a global feature extraction branch, a local feature extraction branch, and a multi-scale feature fusion branch. The global / local feature extraction branch is used to extract global / local features from the i-th scale mirror features using multiple global and local feature extraction paths in different directions. The present invention aims to address the problem of low mirror target detection accuracy in complex backgrounds, where the mirror target is highly consistent with the background, has unclear edges, or is difficult to detect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a mirror detection method and system with cross-scale and multi-directional feature fusion enhancement. Background Art

[0002] Specular detection uses computer vision techniques to identify specular highlights or reflective objects in images or videos. Specular detection is a key task in computer vision, with significant applications in scenarios such as autonomous driving, industrial quality inspection, and security monitoring. In real-world environments, specular reflections often result from strong reflections of light sources from smooth surfaces (such as glass, water, and metal). Their high brightness, lack of texture detail, and blurred boundaries make them easily confused with real objects, leading to errors in target recognition and scene understanding. Traditional methods rely on color space conversion (such as HSV to separate specular highlights) or physical reflectance model analysis, but these methods lack robustness in complex lighting or dynamic scenes. Deep learning-based methods, through end-to-end feature extraction, can learn the essential differences between specular and diffuse reflections. Current research focuses on multimodal data fusion (such as combining RGB-D cameras with polarization sensors), reflectance component separation (reconstructing reflection-free images using generative adversarial networks), and multi-scale attention mechanisms (localizing subtle reflective areas). However, in aerial photography or mobile robot scenarios, mirrored areas can dynamically deform due to changes in viewing angle, and small reflection points (such as car window reflections) are easily drowned out by background noise. Furthermore, phenomena such as the fluctuating reflections of water surfaces and random reflections in rainy and snowy weather in natural environments further complicate detection. Therefore, effectively leveraging the local and global features of images at different scales to accurately capture the complex semantic connections between mirrored and non-mirror areas and achieve high-precision detection of mirrored areas in complex environments has become a key technical challenge that needs to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems in the prior art, a mirror detection method and system with cross-scale multi-directional feature fusion enhancement are provided. The present invention aims to solve the problem of low detection accuracy of mirror targets in complex backgrounds due to the high consistency between the mirror target and the background, unclear edges or difficulty in detection.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A mirror detection method with cross-scale multi-directional feature fusion enhancement includes the following steps: extracting mirror features of M scales from an input image using an encoder, then extracting mirror edge features using an edge extraction network, extracting enhanced features using feature enhancement blocks (HFFs) of M layers, upsampling and fusing the enhanced features at different scales, and decoding them together with the mirror edge features using a decoding head to obtain mirror detection results; the feature enhancement block (HFF) of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch, and a multi-scale feature fusion branch, wherein the global feature extraction branch is used to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions, the local feature extraction branch is used to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions, and the multi-scale feature fusion branch is used to fuse the enhanced features output by the feature enhancement block (HFF) of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block (HFF) of the i-1th layer.

[0006] Optionally, the global feature extraction branch extracts global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions, including: extracting the mirror features of the i-th scale Use the global attention module to extract the global feature map , the features Rotate the specified angle to get the rotation global feature map ; The global feature map , rotate the global feature map The relationship feature extractor is used to capture the continuity and directional correlation features of the target edge based on the eight-way spatial offset mechanism as the relationship feature map, and the relationship feature map is subjected to two pooling feature maps of global maximum pooling and global average pooling respectively. The shared multi-layer perceptron MLP is used to extract features and then sum them to obtain the sum feature map; the global feature map is respectively , rotate the global feature map The sum feature map is normalized by the Sigmoid function to obtain the channel attention map; the global feature map Multiplying the channel attention map with the input feature map to obtain the first global feature map with the same spatial size and the same number of channels , the global feature map will be rotated The second global feature map is recovered by multiplying it with its channel attention map and rotating it inversely by a specified angle to obtain the same spatial size as the input feature map and the same number of channels. .

[0007] Optionally, the method of using a relational feature extractor to capture the continuity and directionality associated features of the target edge as a relational feature map based on an eight-directional spatial offset mechanism includes: based on a predefined set of offsets in eight directions:

[0008] {(0,1),(0,-1),(-1,0),(1,0),(-1,-1),(-1,1),(1,-1),(1,1)},

[0009] A translation operation is performed on the input feature map of the relation feature extractor in each direction, and the point-by-point product of the central pixel and the neighboring pixels is calculated as the relation score to obtain a single-channel relation map in each direction, where each set of offsets in the offset set represents the offset in the x-direction and the y-direction respectively. The single-channel relation maps in eight directions are mapped to the feature space through 1×1 convolution and weighted fusion is performed by applying a learnable directional weight coefficient to obtain the continuity and directional association features of the target edge.

[0010] Optionally, the local feature extraction branch extracts the mirror features of the i-th scale using multiple local feature extraction paths in different directions. Extracting local features includes: extracting the mirror features of the i-th scale using multiple local feature extraction paths in different directions. Use the local attention module to extract the local feature map , the local feature map Rotate the specified angle to get the rotated local feature map ; Local feature map Through two different convolution paths, convolution operation, batch normalization and Sigmoid function normalization are performed, and the difference of the calculation results of the two different convolution paths is obtained as the first local feature of the output ; Rotate the local feature map Convolution operation, batch normalization and Sigmoid function normalization are performed through two different convolution paths. The convolution operation in the two different convolution paths is a dilated convolution operation. The dilated convolution operation in one convolution path uses a 3×3 convolution kernel with a step size of 1, and the dilated convolution operation in the other convolution path uses a 3×3 convolution kernel with a step size of 2. The difference between the calculation results of the two different convolution paths is obtained to obtain the difference feature, and then the difference feature is reversely rotated by a specified angle to recover the second local feature of the output. .

[0011] Optionally, the multi-scale feature fusion branch fuses the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-1th layer, including: processing the enhanced features output by the feature enhancement block HFF of the i-1th layer through convolution and average pooling, wherein the convolution is 1×1 convolution for adjusting the number of channels to twice its own, and the average pooling is used to adjust the dimension to 1 / 2 of the original, and then the enhanced features are combined with the global feature map extracted from the global feature extraction branch. , local feature map extracted from the local feature extraction branch The concatenated features are spliced ​​according to the channel dimension, and then convolution operation, layer normalization and GELU function activation are performed on the concatenated features, where the convolution is 1×1 convolution, and then combined with the global features extracted from the global feature extraction branch. and , local features extracted by local feature extraction branch and The enhanced features of the feature enhancement block HFF of the i-th layer are obtained by splicing according to the channel dimension and then adjusting the number of channels through a multi-layer perceptron MLP. The multi-layer perceptron MLP inputs the input features into a 3×3 convolution kernel to extract features, and then uses 1×1 convolution to expand the number of channels to 4 times the original. A 1×1 convolution is used to compress the channel, and the number of channels is restored. After activation using the GeLU activation function, the enhanced features are normalized using the batch normalization function to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer.

[0012] Optionally, the use of the edge extraction network to extract mirror edge features includes: respectively passing some or all of the mirror features in M ​​scales through a convolution path, respectively undergoing convolution and ReLU activation to extract edge detail feature maps, splicing the edge detail feature maps extracted by all convolution paths according to the channel dimension, and obtaining the mirror edge features through the feature association enhancement module RCAB, wherein the feature association enhancement module RCAB includes a convolution layer, a batch normalization layer, a ReLU activation function, a convolution layer, a batch normalization layer, a channel attention layer CALayer and a jump connection connected in sequence, wherein the jump connection is used to add the input features of the feature association enhancement module RCAB and the output features of the channel attention layer CALayer as the output features of the feature association enhancement module RCAB, and the channel attention layer CALayer includes an average pooling layer, a convolution layer, a ReLU activation function, a convolution layer, a Sigmoid activation function and a channel-by-channel multiplication module connected in sequence, and the channel-by-channel multiplication module is used to multiply the input features of the channel attention layer CALayer and the attention weights of each channel output by the Sigmoid activation function channel by channel as the output features of the channel attention layer CALayer.

[0013] In addition, the present invention also provides a mirror detection system with cross-scale multi-directional feature fusion enhancement, comprising:

[0014] An encoder program unit, configured to extract mirror features of M scales from an input image;

[0015] An edge extraction network program unit is used to extract mirror edge features from mirror features of M scales;

[0016] A feature enhancement program unit is configured to extract enhanced features from mirror features of M scales using M layers of feature enhancement blocks HFF, wherein the feature enhancement block HFF of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch, and a multi-scale feature fusion branch, wherein the global feature extraction branch is configured to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions, wherein the global feature extraction path includes using a relational feature extractor to capture continuity and directional association features of target edges based on an eight-directional spatial offset mechanism, the local feature extraction branch is configured to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions, and the multi-scale feature fusion branch is configured to fuse the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer;

[0017] A sub-scale fusion program unit, used for fusing enhanced features at different scales;

[0018] The decoding head program unit is used to decode the single-channel preliminary mirror area prediction map and the mirror edge features to obtain the mirror detection result.

[0019] In addition, the present invention also provides a mirror detection system with cross-scale multi-directional feature fusion enhancement, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the mirror detection method with cross-scale multi-directional feature fusion enhancement.

[0020] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cross-scale multi-directional feature fusion enhanced mirror detection method through a processor.

[0021] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cross-scale multi-directional feature fusion enhanced mirror detection method through a processor.

[0022] Compared with the prior art, the present invention can mainly achieve the following beneficial effects: the present invention comprises extracting mirror features of M scales from an input image using an encoder, then extracting mirror edge features using an edge extraction network, extracting enhanced features using M layers of feature enhancement blocks HFF, upsampling and fusion of the enhanced features at different scales, and decoding them together with the mirror edge features using a decoding head to obtain mirror detection results; the feature enhancement block HFF of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch, and a multi-scale feature fusion branch; the global feature extraction branch is used to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions; a relational feature extractor is used for extracting features from multi-directional contrast features, and the continuity and directional correlation features of the target edge are captured by the relational feature extractor based on an eight-directional spatial offset mechanism. It effectively solves the edge breakage problem caused by traditional unidirectional convolution and significantly improves the ability to depict curved and irregular mirror boundaries. The local feature extraction branch is used to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions. The multi-scale feature fusion branch is used to fuse the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer, thereby solving the problem of low mirror target detection accuracy caused by the high consistency between the mirror target and the background, unclear edges or difficulty in detection under complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0024] Figure 2 Schematic diagram of the network structure of the feature enhancement block HFF of the i-th layer in an embodiment of the present invention.

[0025] Figure 3 Schematic diagram of the network structure of the edge extraction network in an embodiment of the present invention.

[0026] Figure 4 Schematic diagram of the network structure of the feature correlation enhancement module RCAB in an embodiment of the present invention.

[0027] Figure 5 Schematic diagram of the network structure of the channel attention layer CALayer in an embodiment of the present invention.

[0028] Figure 6 Schematic diagram of the structure of a decoding head in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0030] like Figure 1 As shown, this embodiment provides a mirror detection method with cross-scale multi-directional feature fusion enhancement, including the following steps: using an encoder to extract mirror features of M scales from an input image, then using an edge extraction network to extract mirror edge features, and using M layers of feature enhancement blocks HFF to extract enhanced features, upsampling and fusing the enhanced features at different scales, and then decoding them together with the mirror edge features using a decoding head to obtain a mirror detection result.

[0031] In this embodiment, the input image is recorded as , input image Before using the encoder to extract the mirror features of M scales, the original image is also preprocessed to obtain the input image: the original image is converted into an RGB three-channel matrix to obtain the image input ; Input the image Reshape or resize the input image to the specified size For example, in this embodiment, an image pyramid is constructed to obtain features of M sampling sizes. It should be noted that interpolation is a well-known data processing method. The required interpolation algorithm can be used as needed. For example, the bilinear interpolation algorithm is used in this embodiment. The bilinear interpolation algorithm is an interpolation method based on the weighted average of known pixel values ​​in a local area. It takes into account the changes in pixel values ​​in the horizontal and vertical directions, so that it can better preserve the details and quality of the image. Specifically, bilinear interpolation will find the four pixels closest to the target pixel position in the original image, and then perform weighted average based on the pixel values ​​of these four pixels to obtain the value of the target pixel. First, interpolation is implemented on the X-axis, and then along the X-axis. , Find two points with the same horizontal coordinate as the insertion point and :

[0032] ,

[0033] ,

[0034] In the above formula, and is the interpolation result, 、 、 and is the function value of the corresponding point, and the coordinates of the point to be inserted are , The point coordinates are , The point coordinates are , ~ These are four points near the insertion point, with coordinates corresponding to Then interpolate the Y axis and use the interpolation value from the previous step. and , linear interpolation is performed in the vertical direction.

[0035] ,

[0036] in, This is the value of the insertion point obtained by final interpolation. For example, in this embodiment, the input image is scaled to the specified size of 3×352×352, indicating that the number of channels is 3 and the image size is 352×352, and the input image is obtained. .

[0037] In this embodiment, the input image The feature maps of M scales extracted using the pre-trained encoder are expressed as , as an optional implementation, such as Figure 1 As shown, in this embodiment The value is 4, and the encoder uses the existing PvtNet_v2 encoder. Figure 1 The four boxes inside the encoder represent four coding layers, and the four coding layers output feature maps of four scales: ~ , which can be expressed as a set , and there is:

[0038] ,

[0039] Among them, the subscript Indicates the Layer scale, Indicates the The number of channels at the layer scale, and Represent the height and width of the feature map respectively, and The value is 352.

[0040] like Figure 2As shown, in this embodiment, the feature enhancement block HFF of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch and a multi-scale feature fusion branch. The global feature extraction branch is used to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions. The global feature extraction path includes using a relational feature extractor to capture the continuity and directional correlation features of the target edge based on an eight-way spatial offset mechanism. The local feature extraction branch is used to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions. The multi-scale feature fusion branch is used to fuse the enhanced features output by the feature enhancement block HFF of the i-1-th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer.

[0041] like Figure 2 As shown, in this embodiment, the global feature extraction branch extracts the global features of the mirror features of the i-th scale using multiple global feature extraction paths in different directions, including: extracting the mirror features of the i-th scale Use the global attention module to extract the global feature map , whose shape is [B, C, H, W], where B, C, H, and W are batch size, number of channels, height, and width respectively. The global attention module consists of a concatenated layer normalization and a multi-head attention block with a spatial reduction to 0. Rotate the specified angle to get the rotation global feature map ; The global feature map , rotate the global feature map The relational feature extractor is used to capture the continuity and directional correlation features of the target edge based on the eight-way spatial offset mechanism as the relational feature map, and the relational feature map is subjected to two pooling feature maps of global maximum pooling and global average pooling respectively. The shared multi-layer perceptron MLP is used to extract features (f11~f14) and then sums them to obtain the summed feature map; the global feature map is respectively , rotate the global feature map The sum feature map is normalized by the Sigmoid function to obtain the channel attention map; the global feature map Multiplying the channel attention map with the input feature map to obtain the first global feature map with the same spatial size and the same number of channels , the global feature map will be rotated The second global feature map is recovered by multiplying it with its channel attention map and rotating it inversely by a specified angle to obtain the same spatial size as the input feature map and the same number of channels. . When the shared multi-layer perceptron MLP extracts features, it first uses a 1×1 convolution block to reduce the number of feature channels to 1 / 16 of the original number, uses the ReLU activation function, and then uses a 1×1 convolution block to restore the number of feature channels to obtain the output features. It should be noted that the rotation and reverse rotation in the global feature extraction branch select the required direction and angle as needed. For example, as an optional implementation, in this embodiment, the rotation is a 90-degree clockwise rotation, and the reverse rotation is a 90-degree counterclockwise rotation.

[0042] This embodiment designs a relational feature extractor for multi-directional contrast feature extraction. This extractor captures the continuity and directional correlation features of target edges based on an eight-directional spatial offset mechanism, effectively resolving the edge breakage problem caused by traditional single-directional convolution and significantly improving the ability to depict curved and irregular mirror boundaries. In this embodiment, the use of the relational feature extractor to capture the continuity and directional correlation features of target edges based on an eight-directional spatial offset mechanism as a relational feature map includes: Based on a predefined set of offsets in eight directions:

[0043] {(0,1),(0,-1),(-1,0),(1,0),(-1,-1),(-1,1),(1,-1),(1,1)},

[0044] A translation operation is performed on the input feature map of the relational feature extractor in each direction (the excess part is padded with 0). The above offset set defines a learnable direction weight with a shape of [1, 8]. The point-by-point product of the center pixel and the neighboring pixels is calculated as the relationship score to obtain a single-channel relationship map in each direction, which has a shape of [B, 1, H, W]. Each set of offsets in the offset set represents the offset in the x-direction and the y-direction respectively. The single-channel relationship map in eight directions is mapped to the feature space through 1×1 convolution and the learnable direction weight coefficient is applied for weighted fusion to obtain the continuity and directional association features of the target edge.

[0045] like Figure 2 As shown, in this embodiment, the local feature extraction branch extracts the mirror features of the i-th scale using multiple local feature extraction paths in different directions. The extraction of local features includes: extracting the mirror features of the i-th scale Use the local attention module to extract the local feature map , the local feature map Rotate the specified angle to get the rotated local feature map ; Local feature map Through two different convolution paths (expressed as Conv, BN, Sigmoid), convolution operation (Conv), batch normalization (BN) and Sigmoid function normalization are performed, and the difference of the calculation results of the two different convolution paths is obtained as the first local feature of the output , which can be expressed as:

[0046] ,

[0047] in, and They are the calculation results of two different convolution paths respectively; the local feature map is rotated Convolution operation, batch normalization (LN) and Sigmoid function normalization are performed through two different convolution paths (Conv, LN, GELU). The convolution operation in the two different convolution paths is a dilated convolution operation. The dilated convolution operation in one convolution path uses a 3×3 convolution kernel with a step size of 1, and the dilated convolution operation in the other convolution path uses a 3×3 convolution kernel with a step size of 2. The difference between the calculation results of the two different convolution paths is obtained to obtain the difference feature, and then the difference feature is reversely rotated by a specified angle to recover the second local feature of the output. It should be noted that the rotation and reverse rotation in the local feature extraction branch can be performed in the desired direction and angle as needed. For example, as an optional implementation, in this embodiment, the rotation is a 90-degree clockwise rotation, and the reverse rotation is a 90-degree counterclockwise rotation. This can be the same as or different from the global feature extraction branch.

[0048] like Figure 2 As shown, in this embodiment, the multi-scale feature fusion branch fuses the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-1th layer, including: the enhanced features output by the feature enhancement block HFF of the i-1th layer are processed by convolution and average pooling, wherein the convolution is 1×1 convolution for adjusting the number of channels to twice its own, and the average pooling is used to adjust the dimension to 1 / 2 of the original, and then combined with the global feature map extracted from the global feature extraction branch. , local feature map extracted from the local feature extraction branch The concatenated features are then convolved, layer normalized (LayerNorm) and activated with the GELU function, where the convolution is 1×1 convolution, and then combined with the global features extracted from the global feature extraction branch. and , local features extracted by local feature extraction branch and The enhanced features of the feature enhancement block HFF of the i-th layer are obtained by splicing according to the channel dimension and then adjusting the number of channels through a multi-layer perceptron MLP. The multi-layer perceptron MLP inputs the input features into a 3×3 convolution kernel to extract features, and then uses 1×1 convolution to expand the number of channels to 4 times the original. A 1×1 convolution is used to compress the channel, and the number of channels is restored. After activation using the GeLU activation function, the enhanced features of the feature enhancement block HFF of the i-th layer are obtained by normalization using the batch normalization function (BatchNorm). .

[0049] like Figure 3 As shown, in this embodiment, the use of the edge extraction network to extract mirror edge features includes: passing some or all of the mirror features in M ​​scales through a convolution path, respectively, and then performing convolution and ReLU activation to extract edge detail feature maps. The edge detail feature maps extracted by all convolution paths are then spliced ​​according to the channel dimension and then passed through the feature correlation enhancement module RCAB to obtain the mirror edge features. For example, as an optional implementation, in this embodiment, the feature maps of size 64, 128, 512 are used. 、 、 The edge detail feature maps are extracted through convolution and ReLU activation respectively through a convolution path. The convolution path includes convolution and ReLU activation. The convolution adopts 3×3 convolution with a step size of 1. When the edge detail feature maps extracted by all convolution paths are spliced ​​according to the channel dimension, including 、 Perform bilinear interpolation upsampling The size is used to achieve multi-scale feature alignment. The above upsampling is the basic operation of splicing according to the channel dimension. After splicing according to the channel dimension, the feature , the features The mirror edge features are obtained through the feature association enhancement module RCAB.

[0050] like Figure 4 As shown, in this embodiment, the feature association enhancement module RCAB includes a convolution layer, a batch normalization (BatchNorm) layer, a ReLU activation function, a convolution layer, a batch normalization layer, a channel attention layer CALayer and a jump connection connected in sequence. The jump connection is used to add the input features of the feature association enhancement module RCAB and the output features of the channel attention layer CALayer as the output features of the feature association enhancement module RCAB to achieve adaptive feature enhancement.

[0051] like Figure 5As shown, in this embodiment, the channel attention layer CALayer includes an average pooling layer, a convolution layer, a ReLU activation function, a convolution layer, a Sigmoid activation function and a channel-by-channel multiplication module connected in sequence. The channel-by-channel multiplication module is used to multiply the input features of the channel attention layer CALayer and the attention weights of each channel output by the Sigmoid activation function channel by channel as the output features of the channel attention layer CALayer. After the input features are input into the average pooling layer, the number of channels is reduced to 1 / 16 of the original through the convolution layer using a 1×1 convolution with a step size of 0. After the result is activated by the ReLU activation function, a 1×1 convolution with a step size of 0 is used to restore the number of channels. Finally, it is normalized by the Sigmoid activation function and then dot-multiplied with the input features, thereby realizing the calculation of the attention weights of each channel. .

[0052] like Figure 1 As shown, in this embodiment, when the enhanced features are upsampled and fused at different scales, the enhanced features of the previous layer are upsampled twice (×2) and then fused with the enhanced features of the current layer (channel dimension connection).

[0053] like Figure 6 As shown, in this embodiment, the enhanced features are subjected to scaled upsampling and fusion, and the features are decoded together with the mirror edge features using a decoding head to obtain the mirror detection results, including:

[0054] Step 1: After upsampling the edge features extracted by the edge feature extractor, a 3×3 convolution with a step size of 1 is used to reduce the number of channels to 1 to obtain a single-channel edge region prediction map. ; The output features of the last layer feature enhancement block HFF Upsampling to obtain a single-channel preliminary mirror area prediction map ;

[0055] Step 2: In the first branch, the output features of the last layer feature enhancement block HFF are Convolution operations are performed through three convolution paths respectively; the output features of the convolution operation on the first convolution path are combined with the single-channel preliminary mirror area prediction map Perform element-by-element multiplication as the output of the first convolution path; combine the output features of the convolution operation on the second convolution path with the single-channel preliminary mirror area prediction map Perform element-by-element multiplication, transpose, and perform matrix multiplication on the output of the first convolution path as the output of the second convolution path; perform matrix multiplication on the output features of the convolution operation on the third convolution path and the output of the second convolution path, and then obtain the output of the first branch through a convolution operation;

[0056] In the second branch, the mirror feature Convolution operations are performed through three convolution paths respectively; the output features of the convolution operation on the first convolution path are combined with the single-channel edge area prediction map Perform element-by-element multiplication as the output of the first convolution path; combine the output features of the convolution operation on the second convolution path with the single-channel edge area prediction map Perform element-by-element multiplication, transpose, and then perform matrix multiplication with the output of the first convolution path as the output of the second convolution path; perform matrix multiplication on the output features of the convolution operation on the third convolution path and the output of the second convolution path, and then obtain the output of the second branch through a convolution operation;

[0057] Step 3: Divide the output of the second branch into two paths. The first path performs a matrix multiplication operation with the output of the first branch after convolution and transposition. The second path performs a matrix multiplication operation with the output of the first branch through convolution to obtain the final refined features. , the final refined features Upsampling is performed to obtain the mirror detection result, i.e., the mirror area map.

[0058] The encoder, edge extraction network, feature enhancement module (using the M-layer feature enhancement block HFF to extract enhanced features and perform scaled upsampling and fusion on the enhanced features) and decoding head in this embodiment together form an end-to-end network. The training loss function used in the training of this end-to-end network is The function expression is:

[0059] ,

[0060] in, is the edge detection loss, is the saliency detection loss before fusing edge features, is the salient feature loss after fusing edge features, is the foreground feature similarity loss, is the foreground-background similarity loss, is the background similarity loss, and:

[0061] ,

[0062] ,

[0063] ,

[0064] ,

[0065] ,

[0066] ,

[0067] in, and is the weight parameter, is the binary cross entropy loss, is a single-channel edge region prediction map, is the real edge annotation, The refined features The upsampled mirror saliency feature map, For real significant marking, is the smoothness loss, Characterized by The upsampled mirror saliency feature map, The loss function for similarity comparison is designed to maximize the similarity between two sets of features. is the foreground feature vector, is the background feature vector, The loss function is learned for similarity comparison, aiming at the similarity of two sets of features.

[0068] ,

[0069] ,

[0070] SimMin( )= cos( )

[0071] SimMax( )= cos( )

[0072] ,

[0073] in, for The i-th row and j-th column feature of for Single channel feature map after upsampling, for The channel in row i and column j, is the fusion feature output by the decoding head, For extremely small constants, here we use 1e-8; represents 1×1 convolution, is the feature association enhancement module, is the feature in the decoding head before being spliced ​​with the edge feature. " is the splicing operation, represents a 3×3 convolution, It is a single channel edge area prediction map.

[0074] In order to verify the effectiveness of the mirror detection method enhanced by cross-scale multi-directional feature fusion in this embodiment, this embodiment conducts model training and testing on the WSMD (Weakly-Supervised Mirror Detection) dataset, and conducts comparative analysis with existing methods. The WSMD dataset is the first weakly supervised mirror detection dataset based on graffiti annotation, which is designed to solve the problems of high annotation cost and strong reflection interference in mirror area detection. The WSMD dataset integrates multiple public mirror detection datasets (MSD, PMD, Mirror-RGBD), which contains a total of 10,158 training sets and test sets. In order to verify the effectiveness of the method of this embodiment, the existing fully supervised mirror detection model MirrorNet is selected for comparison, and the following indicators are used to comprehensively evaluate the model performance: structural similarity measurement ( , the bigger the better), enhanced metrics ( , the larger the better), the harmonic mean of the maximum recall rate ( FW , the larger the better), mean absolute error ( , the smaller the better), the final results are shown in Table 1.

[0075] Table 1 Comparison of recognition results between the method in this embodiment and the MirrorNet method on the WSMD dataset

[0076]

[0077] As can be seen from Table 1, the structural similarity metric of the hidden target segmentation method of this embodiment ( ), enhanced metrics ( ), the harmonic mean of maximum recall ( FW ), mean absolute error ( ) are higher than the existing fully supervised mirror detection model SETR. It can be seen that the mirror detection method enhanced by cross-scale multi-directional feature fusion in this embodiment can solve the problem of low detection accuracy of mirror targets under complex backgrounds due to the high consistency between the mirror target and the background, unclear edges or difficulty in detection. In particular, this embodiment designs a relational feature extractor for multi-directional contrast feature extraction. The relational feature extractor captures the continuity and directional correlation features of the target edge based on the eight-way spatial offset mechanism, effectively solves the edge breakage problem caused by traditional unidirectional convolution, and significantly improves the ability to depict curved and irregular mirror boundaries; by using the local feature extraction branch to calculate the difference between the calculation results of two different convolution paths, it can realize difference contrast learning to enhance the discriminative representation of the target and the background, and enhance the semantic difference between the mirror and non-mirror areas. The HFF module introduces a dynamic modeling mechanism of spatial relations, fuses local details and global semantic information in the feature disentanglement process, aggregates multi-scale spatial context through adaptive weight distribution, and enhances the detection stability of small targets and targets with variable sizes. By introducing a global rotation feature branch and a local rotation feature branch, rotation invariance constraints and bidirectional attention interaction are realized, multi-view feature alignment and cross-modal information complementation are achieved, complex background interference is effectively suppressed, and the semantic consistency of the target main area is enhanced.

[0078] In addition, this embodiment also provides a mirror detection system with cross-scale and multi-directional feature fusion enhancement, including:

[0079] An encoder program unit, configured to extract mirror features of M scales from an input image;

[0080] An edge extraction network program unit is used to extract mirror edge features from mirror features of M scales;

[0081] A feature enhancement program unit is configured to extract enhanced features from mirror features of M scales using M layers of feature enhancement blocks HFF, wherein the feature enhancement block HFF of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch, and a multi-scale feature fusion branch, wherein the global feature extraction branch is configured to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions, wherein the global feature extraction path includes using a relational feature extractor to capture continuity and directional association features of target edges based on an eight-directional spatial offset mechanism, the local feature extraction branch is configured to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions, and the multi-scale feature fusion branch is configured to fuse the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer;

[0082] A sub-scale fusion program unit, used for fusing enhanced features at different scales;

[0083] The decoding head program unit is used to decode the single-channel preliminary mirror area prediction map and the mirror edge features to obtain the mirror detection result.

[0084] In addition, this embodiment also provides a mirror detection system with cross-scale multi-directional feature fusion enhancement, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the mirror detection method with cross-scale multi-directional feature fusion enhancement.

[0085] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cross-scale multi-directional feature fusion enhanced mirror detection method through a processor.

[0086] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cross-scale multi-directional feature fusion enhanced mirror detection method through a processor.

[0087] Those skilled in the art should understand that the technical solution provided by the present invention may be in the form of a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0088] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A mirror detection method enhanced by cross-scale multi-directional feature fusion, characterized by: The method comprises the following steps: extracting mirror features of M scales from an input image using an encoder, extracting mirror edge features using an edge extraction network, extracting enhanced features using a feature enhancement block HFF of M layers, performing scale-up sampling and fusion on the enhanced features, and decoding the enhanced features together with the mirror edge features using a decoding head to obtain a mirror detection result; the feature enhancement block HFF of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch and a multi-scale feature fusion branch, wherein the global feature extraction branch is used to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions, wherein the global feature extraction path includes using a relational feature extractor to capture the continuity and directional correlation features of the target edge based on an eight-way spatial offset mechanism, the local feature extraction branch is used to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions, and the multi-scale feature fusion branch is used to fuse the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer.

2. The mirror detection method based on cross-scale multi-directional feature fusion enhancement according to claim 1, characterized in that: The global feature extraction branch extracts the mirror features of the i-th scale using multiple global feature extraction paths in different directions. The global feature extraction includes: extracting the mirror features of the i-th scale Use the global attention module to extract the global feature map , the features Rotate the specified angle to get the rotation global feature map ; The global feature map , rotate the global feature map The relationship feature extractor is used to capture the continuity and directional correlation features of the target edge based on the eight-way spatial offset mechanism as the relationship feature map, and the relationship feature map is subjected to two pooling feature maps of global maximum pooling and global average pooling respectively. The shared multi-layer perceptron MLP is used to extract features and then sum them to obtain the sum feature map; the global feature map is respectively , rotate the global feature map The sum feature map is normalized by the Sigmoid function to obtain the channel attention map; the global feature map Multiplying the channel attention map with the input feature map to obtain the first global feature map with the same spatial size and the same number of channels , the global feature map will be rotated The second global feature map is recovered by multiplying it with its channel attention map and rotating it inversely by a specified angle to obtain the same spatial size as the input feature map and the same number of channels. .

3. The mirror detection method based on cross-scale multi-directional feature fusion enhancement according to claim 2 is characterized in that: The use of the relational feature extractor to capture the continuity and directionality associated features of the target edge based on the eight-way spatial offset mechanism includes: based on a predefined set of offsets in eight directions: {(0,1),(0,-1),(-1,0),(1,0),(-1,-1),(-1,1),(1,-1),(1,1)}， A translation operation is performed on the input feature map of the relation feature extractor in each direction, and the point-by-point product of the central pixel and the neighboring pixels is calculated as the relation score to obtain a single-channel relation map in each direction, where each set of offsets in the offset set represents the offset in the x-direction and the y-direction respectively. The single-channel relation maps in eight directions are mapped to the feature space through 1×1 convolution and weighted fusion is performed by applying a learnable directional weight coefficient to obtain the continuity and directional association features of the target edge.

4. The mirror detection method based on cross-scale multi-directional feature fusion enhancement according to claim 2, characterized in that: The local feature extraction branch extracts the mirror features of the i-th scale into multiple local feature extraction paths in different directions. The local feature extraction includes: extracting the mirror features of the i-th scale into Use the local attention module to extract the local feature map , the local feature map Rotate the specified angle to get the rotated local feature map ; Local feature map Through two different convolution paths, convolution operation, batch normalization and Sigmoid function normalization are performed, and the difference of the calculation results of the two different convolution paths is obtained as the first local feature of the output ; Rotate the local feature map Convolution operation, batch normalization and Sigmoid function normalization are performed through two different convolution paths. The convolution operation in the two different convolution paths is a dilated convolution operation. The dilated convolution operation in one convolution path uses a 3×3 convolution kernel with a step size of 1, and the dilated convolution operation in the other convolution path uses a 3×3 convolution kernel with a step size of 2. The difference between the calculation results of the two different convolution paths is obtained to obtain the difference feature, and then the difference feature is reversely rotated by a specified angle to recover the second local feature of the output. .

5. The mirror detection method based on cross-scale multi-directional feature fusion enhancement according to claim 4 is characterized in that: The multi-scale feature fusion branch fuses the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-1th layer. The enhanced features output by the feature enhancement block HFF of the i-1th layer are processed by convolution and average pooling, where the convolution is 1×1 convolution, which is used to adjust the number of channels to twice its own, and the average pooling is used to adjust the dimension to 1 / 2 of the original, and then combined with the global feature map extracted from the global feature extraction branch. , local feature map extracted from the local feature extraction branch The concatenated features are spliced ​​according to the channel dimension, and then convolution operation, layer normalization and GELU function activation are performed on the concatenated features, where the convolution is 1×1 convolution, and then combined with the global features extracted from the global feature extraction branch. and , local features extracted by local feature extraction branch and The enhanced features of the feature enhancement block HFF of the i-th layer are obtained by splicing according to the channel dimension and then adjusting the number of channels through a multi-layer perceptron MLP. The multi-layer perceptron MLP inputs the input features into a 3×3 convolution kernel to extract features, and then uses 1×1 convolution to expand the number of channels to 4 times the original. A 1×1 convolution is used to compress the channel, and the number of channels is restored. After activation using the GeLU activation function, the enhanced features are normalized using the batch normalization function to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer.

6. The mirror detection method based on cross-scale multi-directional feature fusion enhancement according to claim 1, characterized in that: The method of extracting mirror edge features using an edge extraction network includes: respectively passing some or all of the mirror features in M ​​scales through a convolution path, respectively undergoing convolution and ReLU activation to extract edge detail feature maps, splicing the edge detail feature maps extracted by all convolution paths according to the channel dimension, and obtaining the mirror edge features through a feature association enhancement module RCAB, wherein the feature association enhancement module RCAB includes a sequentially connected convolution layer, a batch normalization layer, a ReLU activation function, a convolution layer, a batch normalization layer, a channel attention layer CALayer, and a jump connection, wherein the jump connection is used to add the input features of the feature association enhancement module RCAB and the output features of the channel attention layer CALayer as the output features of the feature association enhancement module RCAB, and the channel attention layer CALayer includes a sequentially connected average pooling layer, a convolution layer, a ReLU activation function, a convolution layer, a Sigmoid activation function, and a channel-by-channel multiplication module, wherein the channel-by-channel multiplication module is used to multiply the input features of the channel attention layer CALayer and the attention weights of each channel output by the Sigmoid activation function channel by channel as the output features of the channel attention layer CALayer.

7. A mirror detection system with cross-scale multi-directional feature fusion enhancement, characterized by: include: An encoder program unit, configured to extract mirror features of M scales from an input image; An edge extraction network program unit is used to extract mirror edge features from mirror features of M scales; A feature enhancement program unit is configured to extract enhanced features from mirror features of M scales using M layers of feature enhancement blocks HFF, wherein the feature enhancement block HFF of any i-th layer is composed of a global feature extraction branch, a local feature extraction branch, and a multi-scale feature fusion branch, wherein the global feature extraction branch is configured to extract global features from the mirror features of the i-th scale using multiple global feature extraction paths in different directions, wherein the global feature extraction path includes using a relational feature extractor to capture continuity and directional association features of target edges based on an eight-directional spatial offset mechanism, the local feature extraction branch is configured to extract local features from the mirror features of the i-th scale using multiple local feature extraction paths in different directions, and the multi-scale feature fusion branch is configured to fuse the enhanced features output by the feature enhancement block HFF of the i-1th layer with the output features of the global feature extraction branch and the local feature extraction branch to obtain the enhanced features output by the feature enhancement block HFF of the i-th layer; A sub-scale fusion program unit, used for fusing enhanced features at different scales; The decoding head program unit is used to decode the single-channel preliminary mirror area prediction map and the mirror edge features to obtain the mirror detection result.

8. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the mirror detection method enhanced by cross-scale multi-directional feature fusion according to any one of claims 1 to 6 through a processor.

9. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the mirror detection method enhanced by cross-scale multi-directional feature fusion according to any one of claims 1 to 6 through a processor.

Images