Mirror image detection method based on generalized implicit reflection perception

Abstract

The invention discloses a mirror image detection method based on generalized implicit reflection perception. A generalized implicit reflection perception framework based on visual prompt is constructed. Specifically, a prompt chain generation module is introduced, mirror image area features are sensed through thinking chain reasoning, and mirror image area feature sensing can be achieved even under the condition that no entity exists. The clear visual prompt is fixed or slightly updated at the image level, and the invention provides a prompt updating module based on a gating mechanism so as to realize feature level and semantic perception updating, reduce reflection interference and enhance a prompt chain. In addition, the invention proposes a cue injection module to drive the model to position the mirror. The modules provided by the invention are all plug-and-play. According to the method, the most advanced performance is realized with extremely low calculation complexity on four mirror image references, and the method exceeds single-mode, multi-mode and video-level methods of complete supervision and weak supervision. Good performance is also obtained on seven pieces of glass, camouflage and underwater benchmarks.

Description

Technical Field

[0001] The present invention belongs to the technical field of mirror detection in image detection, and more specifically, relates to a mirror image detection method based on generalized implicit reflection perception. Background Art

[0002] Mirrors reflect physical entities and create identical images, which can seriously confuse and affect the understanding and modeling of visual space. Some works try to locate the mirror area to eliminate reflection interference. For example, Yang et al., 2019, Lin, Wang, and Lau, 2020, Guan, Lin, and Lau, 2022 designed several components to establish the association between entities and images; He, Lin, and Lau, 2023, Huang et al., 2023a used rotation strategies to build mirror symmetry consistency; Zha et al., 2024 built a weakly supervised model based on graffiti annotation, namely WSMD, and achieved comparable performance to fully supervised methods; Mei et al., 2021; Tan et al., 2022 used the difference between mirror and non-mirror areas to distinguish, namely depth and content distribution, and Lin, Tan, and Lau, 2023, Warren et al., 2024 proposed a video-level method based on motion cues.

[0003] Different from traditional low-level structure segmentation tasks such as Camouflage Object Detection (COD), Salient Object Detection (SOD) or Semantic Segmentation (SS), Mirror Detection (MD) aims to overcome the unique interference from reflective imaging and locate the mirror area. In specific tasks, such as object recognition (Wang et al., 2023b), path planning (Tang and Ma, 2024) and 3D space reconstruction in experience agents (Wu et al., 2024), the importance of clearly distinguishing between images and entities is self-evident.

[0004] There are three main problems with mirror detection: 1) Camouflage: Mirrors are often placed indoors, and their reflective properties make them highly similar to their surroundings. Relying solely on the reflection mechanism to establish an association between an entity and an image can lead to false positives, such as mistakenly mistaking transparent objects such as glass or reflective ceramic walls for mirrors. Conversely, it can also lead to false misses, such as a car rearview mirror that only has an image but no corresponding physical entity. 2) Irregular shapes: Due to occlusion, mirrors take on highly irregular shapes, such as a bouquet of flowers, which makes it difficult for the model to determine through shape matching (such as a circle or rectangle). 3) Large scale variation: The mirror area may occupy the entire image or less than one-tenth, so accurate and complete positioning is challenging.

[0005] Previous work has designed custom modules to explicitly establish reflection and scale perception to address the above issues. Existing mirror detection frameworks can be roughly divided into three categories. a) The naive U-shaped encoder-decoder architecture represented by HetNet (He, Lin, and Lau, 2023) captures details and semantic correlations through high- and low-level designs of several components. b) The Siamese encoding architecture represented by SATNet (Huang et al., 2023a) rotates random images and designs a two-stream network to explicitly capture symmetric representation consistency. c) The explicit visual cue-assisted architecture represented by EVP (Liu et al., 2023) and VSCode (Luo et al., 2024) generates frequency or depth representations of the input and uses adapters in the encoding stage to adapt to different tasks or domains. Mirror detection aims to segment mirror regions based on the difference between entities and images. Existing mirror detection methods explicitly exploit the symmetry of mirror imaging or additional cues (such as depth) to assist localization. However, in some cases, only images are observed without corresponding entities, which makes it difficult to establish explicit symmetry associations. In addition, auxiliary cues inevitably introduce noise and lack generalization. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a mirror detection method based on generalized implicit reflection perception, which can perceive the characteristics of the mirror area through thought chain reasoning even in the absence of an entity. At the same time, it reduces reflection interference and enhances the prompt chain, obtains the best prompt and injects a feature map to accurately and completely locate the mirror area.

[0007] To achieve the above-mentioned object of the invention, the present invention provides a mirror image detection method based on generalized implicit reflection perception, which is characterized by comprising the following steps:

[0008] (1) Generate multi-scale features Figure X i

[0009] Use the Pyramid Vision Transformer (PVT) network as the backbone network to transform the input image I∈R 3×H×W Extracting multi-scale long-range dependency feature maps generates multi-scale feature maps where i∈{1,2,3,4}, R represents space, C i represents the number of channels of the i-th scale, H and W represent the height and width respectively;

[0010] (2) Gradually aggregate adjacent features to generate feature map S i

[0011] In the feature aggregation module, the multi-scale features Figure X i With X i-1 , i∈{2,3,4} is aggregated to obtain the feature map S i :

[0012] S i =Conv(Concat(UP↑(X i ),X i-1 ))

[0013] Multi-scale features Figure X 4 is aggregated with feature map S2 to obtain feature map S1:

[0014] S1=Conv(Concat(S2,Conv(X4)))

[0015] Among them, UP ↑ means upsampling, Concat means channel-wise connection, and Conv means convolution;

[0016] (3) Decoding generates prediction graph o 3 And update the feature map required for decoding the next scale And prompt parameters

[0017] 3.1) In the aggregation prediction module, the multi-scale features are first Figure X 4 The convolutional feature map Conv(X4) is aggregated with the feature map S4 to obtain the aggregated feature F 3 :

[0018] F 3 =Conv(Concat(Conv(Conv(X4)),Conv(S4)))

[0019] Then, the aggregate feature F 3 Convolution is performed separately to obtain the prediction graph o 3 And the feature map f 3 ;

[0020] 3.2) Generate prompt parameter p 3

[0021] In the prompt chain generation module, the prompt parameters are first randomly initialized

[0022] According to the feature map f 3 Generate channel weights

[0023]

[0024] Generate channel weights

[0025]

[0026] Among them, Conv 1×1 represents 1×1 convolution, σ represents activation function, GAP represents global average pooling, and GMP represents global maximum pooling;

[0027] Finally, the prompt parameter p is generated 3 :

[0028]

[0029] Among them, ⊙ represents element-by-element multiplication;

[0030] 3.3) In the prompt update module, generate updated prompt parameters

[0031]

[0032] Among them, Shuffle means channel shuffle;

[0033] 3.4) Generate feature maps in the prompt injection module And prompt parameters

[0034] In the prompt injection module, first according to the prompt parameters Generate feature map

[0035]

[0036] Among them, Conv 1×1 Represents 1×1 convolution, Conv 1×1 represents 3×3 convolution, and Split represents the division into three feature maps by channel, namely

[0037] In the same way, based on the feature map f 3 Generate feature map

[0038] Then the fusion prediction graph o 3 With feature map Get mirror feature

[0039]

[0040] Then generate the correlation matrix Corr from the prompt to the feature map p→f :

[0041]

[0042] Where τ is a learnable scaling factor;

[0043] At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p :

[0044]

[0045] Finally, we get the feature map with the hint injected And the updated prompt parameters

[0046]

[0047]

[0048] in, Represents matrix multiplication;

[0049] (4) Decoding generates prediction graph o 2 And update the feature map required for decoding the next scale And prompt parameters

[0050] 4.1) In the aggregation prediction module, the feature map Aggregate with feature map S3 to obtain aggregate feature F 2 :

[0051]

[0052] Then, the aggregate feature F 2 Convolution is performed separately to obtain the prediction graph o 2 And the feature map f 2 ;

[0053] 4.2) Generate prompt parameter p 2

[0054] In the prompt chain generation module, the prompt parameters are first Perform 3×3 transposed convolution and upsample by 2 times to get the prompt parameters

[0055] According to the feature map f 2 Generate channel weights

[0056]

[0057] Generate channel weights

[0058]

[0059] Finally, the prompt parameter p is generated 2 :

[0060]

[0061] 4.3) Generate updated prompt parameters

[0062] In the prompt update module, change the prompt parameter Upsampling, aggregation and redistribution of channel features to obtain prompt parameters

[0063]

[0064]

[0065]

[0066] Among them, Γ 1 , Γ 2 is the convolution + GELU activation function with different parameters, α and β are learnable parameters;

[0067] Then, generate high-frequency cue parameters

[0068]

[0069] Among them, Γ 3 It is a convolution + GELU activation function with different parameters;

[0070] Then, frequency aggregation is performed to generate the prompt parameter p′ 3 :

[0071]

[0072] Among them, ζ and γ are learnable parameters;

[0073] Then, the prompt parameter p′ 3 With the prompt parameter p 2 Connect by channel and get the prompt parameter p′ 2 , generate hint denoising parameters

[0074]

[0075] 4.4) Generate feature maps in the prompt injection module And prompt parameters

[0076] In the prompt injection module, first according to the prompt parameters Generate feature map

[0077]

[0078] In the same way, based on the feature map f 2 Generate feature map

[0079] Then use the prediction graph o 2 To decouple feature maps Get feature map

[0080]

[0081] Then generate the correlation matrix Corr from the prompt to the feature map p→f :

[0082]

[0083] At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p :

[0084]

[0085] Finally, we get the feature map with the hint injected And the updated prompt parameters

[0086]

[0087]

[0088] (5) Decoding generates prediction graph o 1 And update the feature map required for decoding the next scale And prompt parameters

[0089] 5.1) In the aggregation prediction module, the feature map Aggregate with feature map S2 to obtain aggregate feature F 1 :

[0090]

[0091] Then, the aggregate feature F 1 Convolution is performed separately to obtain the prediction graph o 1 And the feature map f 1 ;

[0092] 5.2) Generate prompt parameter p 1

[0093] In the prompt chain generation module, the prompt parameters are first Perform 3×3 transposed convolution and upsample by 2 times to get the prompt parameters

[0094] According to the feature map f 1 Generate channel weights

[0095]

[0096] Generate channel weights

[0097]

[0098] Finally, the prompt parameter p is generated 1 :

[0099]

[0100] 5.3) Generate updated prompt parameters

[0101] In the prompt update module, change the prompt parameter Upsampling, aggregation and redistribution of channel features to obtain prompt parameters

[0102]

[0103]

[0104]

[0105] Then, generate high-frequency cue parameters

[0106]

[0107] Then, frequency aggregation is performed to generate the prompt parameter p′ 2 :

[0108]

[0109] Then, the prompt parameter p′ 3 , p′ 2 With the prompt parameter p 1 Connect by channel and get the prompt parameter p′ 1 , generate hint denoising parameters

[0110]

[0111] 5.4) Generate feature maps in the prompt injection module And prompt parameters

[0112] In the prompt injection module, first according to the prompt parameters Generate feature map

[0113]

[0114] In the same way, based on the feature map f 2 Generate feature map

[0115] Then use the prediction graph o 1 To decouple feature maps Get feature map

[0116]

[0117] Then generate the correlation matrix Corr from the prompt to the feature map p→f :

[0118]

[0119] At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p :

[0120]

[0121] Finally, we get the feature map with the hint injected And the updated prompt parameters

[0122]

[0123]

[0124] (6) In the aggregation prediction module, the feature map Aggregate with feature map S1 to obtain aggregate feature F 0 :

[0125]

[0126] Then the aggregate feature F 0 Convolution is performed separately to obtain the prediction graph o 0 And the feature map f 0 ;

[0127] (7) Training a generalized implicit reflection perception network based on visual cues

[0128] The pyramid visual transformer network, feature aggregation module, aggregation prediction module, prompt chain generation module, prompt update module and prompt injection module constitute a generalized implicit reflection perception network based on visual prompts. The labeled image I in the training data set is processed according to steps (1) to (6) to obtain the prediction image o j , j = 0, 1, 2, 3, based on the prediction graph o j Calculate the binary cross entropy loss separately and weighted intersection-over-union loss Get the total loss L under full supervision fully :

[0129]

[0130] Or based on the prediction graph j Calculate partial cross entropy loss separately and smoothing loss Get the total loss L under weak supervision wea :

[0131]

[0132] According to the total loss L fully Or the total loss L weakly Training a generalized implicit reflection perception network based on visual cues;

[0133] (8) Mirror detection

[0134] The image to be detected is sent to the generalized implicit reflection perception network based on visual cues, and processed according to steps (1) to (6) to obtain a refined prediction map o 0 , complete the mirror detection.

[0135] The object of the present invention is achieved in this way.

[0136] The present invention constructs a generalized implicit reflection perception (IRP) framework based on visual cues based on the mirror detection method of generalized implicit reflection perception. Specifically, a prompt chain generation (PCG) module is introduced to perceive the mirror area features through thought chain reasoning, and the mirror area feature perception can be achieved even in the absence of entities. Explicit visual cues are fixed or slightly updated at the image level. The present invention proposes a prompt updating (PU) module based on a gating mechanism to achieve feature level and semantic perception updates, reduce reflection interference and enhance the prompt chain. In addition, the present invention proposes a prompt injection (PI) module to drive the model to locate the mirror. The proposed modules are all plug-and-play. The present invention achieves state-of-the-art performance on four mirror benchmarks with extremely low computational complexity, surpassing fully supervised and weakly supervised single-modal, multi-modal and video-level methods. Good performance is also achieved on seven glass, camouflage and underwater benchmarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Figure 1 It is an overall architecture diagram of a specific implementation method of the image detection method based on generalized implicit reflection perception of the present invention;

[0138] Figure 2 It is a flowchart of a specific implementation method of the image detection method based on generalized implicit reflection perception of the present invention;

[0139] Figure 3 yes Figure 1 The structure diagram of the prompt chain generation module (PCG) shown;

[0140] Figure 4 yes Figure 1 The structure diagram of the prompt update module (PU) based on the gating mechanism is shown;

[0141] Figure 5 yes Figure 1 The structure diagram of the prompt injection module (PI) shown;

[0142] Figure 6 is a diagram of the existing mirror detection and general two-segmentation framework, (a) is a simple encoding-decoding framework represented by HetNet (He, Lin, and Lau 2023), (b) is a dual-stream architecture represented by SATNet, (c) is an explicit visual cue architecture represented by EVP, and (d) is the IRP architecture of the present invention;

[0143] Figure 7The weighted F measurement is the difference between the present invention (IRP) and 10 existing detection methods. IoU and comparison diagram using the MSD dataset (Yang et al., 2019), where smaller circles represent fewer parameters, where (a) is a quantitative comparison and (b) is a result comparison;

[0144] Figure 8 It is a qualitative comparison diagram on the MD dataset;

[0145] Fig. 9 It is the qualitative ablation map of different modules;

[0146] Fig.10 This is a failure scenario diagram. DETAILED DESCRIPTION

[0147] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0148] The IRP framework proposed in the present invention does not require explicit generation of visual cues or adapters. Instead, the present invention learns implicit visual cues in the decoding stage. Frequency and depth cues inevitably introduce noise and are not ideal. In contrast, the implicit cues learned by the present invention have higher-order semantics, effectively avoid noise interference and enhance the representation of mirror areas. In addition, the present invention can be generalized to different tasks, such as COD. Through implicit learning, symmetry perception can be obtained from any angle, while HetNet or SATNet only establish entity-image correlation within a fixed range by randomly rotating 90 degrees (or multiples thereof).

[0149] This paper proposes an IRP framework based on visual cues to implicitly perceive the context of the mirror area. Specifically, inspired by large language models (LLMs) (Wei et al., 2022), this paper proposes a PCG module, which aims to enhance the model's visual reasoning ability by building a series of intermediate cues, namely, a cue chain. This is crucial for establishing associations between entities and images, especially when corresponding entities are missing. However, compared with explicit learning, the representation space of implicit learning is large. In addition, the image can change depending on the environment in which the mirror is located, and irregular occlusions and scale changes further expand the boundaries of the learning space, making it difficult to learn key cues. Therefore, this paper proposes a PU module to reduce the exploration path and alleviate noise interference to obtain the best cue. The present invention further proposes a PI module to inject cues into the feature map to accurately and completely locate the mirror area. Extensive experiments on four mirror benchmarks and seven glass, camouflage and underwater benchmarks demonstrate the superiority and robustness of our method.

[0150] The present invention rethinks the existing MD framework and formulates the IRP framework to implicitly perceive reflections and solve the problems of occlusion and scale variation. The present invention proposes a PCG module to build a hint chain and a PU module to optimize hints. The proposed PI module injects hints into the feature map to accurately and completely locate the mirror area. All proposed modules are plug-and-play. The present invention surpasses various SOTA methods on four MD benchmarks with low computational complexity. And we verify its generalization ability on seven datasets involving glass, camouflage and underwater scenarios.

[0151] Related Work

[0152] Hint learning: Hint learning has been widely used in natural language processing (Wei et al., 2022), recommender systems (Li, Zhang, and Chen, 2023), and open-world perception (Zhu et al., 2023). (Liu et al., 2023) introduced visual cues generated from high-frequency components of the input and applied them to low-level structure segmentation tasks. However, image-based explicit hint construction may introduce additional noise and lack semantic representation, which makes it difficult to establish associations between entities and images or capture the context of mirror areas. Similarly, (Luo et al., 2024) constructed two-dimensional visual hints based on mixed datasets and applied them to the COD task. However, this significantly increases the training cost and lacks generalization ability. In contrast, the present invention constructs semantically-aware implicit hints to reduce reflection interference and achieves superior performance in multiple tasks (such as COD) with lower computational overhead.

[0153] Image Segmentation: Binary segmentation tasks, such as COD and SOD, have some overlapping problems with MD, such as scale variation, but are difficult to handle reflections and object occlusions. Taking SOD as an example, if there are only images in the image, they may be considered as salient objects. In addition, general SS methods, such as SAM (Kirillov et al., 2023), also face these problems. Therefore, it cannot be directly applied to MD tasks.

[0154] Figure 1 It is an overall architecture diagram of a specific implementation method of the image detection method based on generalized implicit reflection perception of the present invention.

[0155] In this embodiment, if Figure 1As shown in the figure, the generalized implicit reflection perception network (architecture) based on visual cues consists of three core components: PCG, PU and PI modules. The encoder is used to extract multi-scale features, continuously upsample and aggregate, and then the PCG module is applied in the decoder to generate a prompt chain for association reasoning between entities and images. The PU module is then further used to enhance the coupling between elements in the chain and reduce noise. Finally, the PI module is used to inject refined cues into the feature map, which is decoupled by using the predicted mask to improve the semantic difference between the mirror and non-mirror areas. Specifically:

[0156] In this example, multi-scale features are obtained through the pyramid visual transformer (Wang et al., 2021) Among them, C i represents the number of channels at the i-th scale, H and W represent the height and width respectively, i∈{1, 2, 3, 4}, and R represents the space. In the decoding stage, the present invention randomly initializes the prompt parameters and generates a prompt chain through the PCG module where j∈{1, 2, 3}, C j =64×(4-j), and then further use the PU module to update and enhance the prompt chain. For the prompt parameter p 3 , the present invention integrates the prompt parameter p 3 And the prompt parameter p after channel shuffling 3 To implicitly learn symmetry consistency. For the hint parameter p 2 and p 1 , the present invention respectively fuses the prompt parameter p′ after noise filtering 3 and p′ 3 , p′ 2 . Finally, the PI module is used to inject the refined hints into the feature map and iterate the hints.

[0157] Figure 2 It is a flowchart of a specific implementation of the mirror image detection method based on generalized implicit reflection perception of the present invention.

[0158] In this embodiment, if Figure 2 As shown, the mirror image detection method based on generalized implicit reflection perception of the present invention includes the following steps:

[0159] Step S1: Generate multi-scale features Figure X i

[0160] In this embodiment, if Figure 1 As shown, the Pyramid Vision Transformer (PVT) network (Wang et al., 2021) is used as the backbone network for the input image I∈R 3×H×WExtracting multi-scale long-range dependency feature maps generates multi-scale feature maps where i∈{1, 2, 3, 4}, R represents space, C i represents the number of channels at the i-th scale, H and W represent the height and width respectively.

[0161] Step S2: Gradually aggregate adjacent features to generate feature map S i

[0162] In this embodiment, if Figure 1 As shown in Figure 2, in the feature aggregation module (FA module), the multi-scale features are Figure X i With X i-1 , i∈{2, 3, 4} is aggregated to obtain the feature map S i :

[0163] S i =Conv(Concat(UP↑(X i ), X i-1 ))

[0164] Multi-scale features Figure X 4 is aggregated with feature map S2 to obtain feature map S1:

[0165] S1=Conv(Concat(S2,Conv(X4)))

[0166] Among them, UP ↑ means upsampling, Concat means channel-wise connection, and Conv means convolution.

[0167] In this embodiment, if Figure 1 As shown in the figure, the aggregation and prediction module (Aggregating and Predictingmodule, referred to as AP module), the prompt chain generation module (Prompt Chain Generation, referred to as PCG module), the prompt updating module (Prompt Updating, referred to as PU) and the prompt injection (Prompt Injection, referred to as PI) module constitute the decoder Decoder layer by layer to obtain the prediction graph o j , j=0,1,2,3, as in steps S3 to S6.

[0168] Step S3: Decode and generate prediction graph o 3 And update the feature map required for decoding the next scale And prompt parameters

[0169] Step S3.1: Figure 1As shown, in the aggregation prediction module, i.e., the AP module, the multi-scale features are first Figure X 4 The convolutional feature map Conv(X4) is aggregated with the feature map S4 to obtain the aggregated feature F 3 :

[0170] F 3 =Conv(Concat(Conv(UP↑(Conv(X4))),Conv(S4)))

[0171] Then, the aggregate feature F 3 Convolution is performed separately to obtain the prediction graph o 3 And the feature map f 3 .

[0172] Step S3.2: Generate prompt parameter p 3

[0173] Unlike explicit prompts that obtain low-level detailed representations in the encoding stage, the present invention aims to capture the contextual information of the mirror area. In the decoding stage, the present invention establishes the association and difference between entities and images through semantic perception, thereby providing a unified solution to the problems in the mirror detection task, namely, reflection interference, occlusion, and learning boundaries wider than the explicit space, which are difficult to achieve spatial modeling through a single prompt. Inspired by the thought chain reasoning in the Large Language Model (LLM), the present invention constructs a visual prompt chain to find the best prompt through multi-step reasoning. Specifically, the present invention constructs a prompt chain generation module (Prompt Chain Generation, PCG module for short), whose structural details are as follows Figure 3 As shown. First, randomly initialize the learnable parameters and chained by transposed convolution (Dumoulin and Visin, 2016) with 2× and 4× upsampling,

[0174] In this embodiment, if Figure 3 As shown in (a), in the prompt chain generation module, i.e., the PCG module, the prompt parameters are first randomly initialized by progressive transposed convolution (Dumoulin and Visin, 2016)

[0175] like Figure 1 , Figure 3 As shown in (a), the present invention will prompt the parameters and f 3Based on the pixel weights, the feature content is dynamically perceived and convergence is accelerated. The present invention applies Global Average Pooling (GAP) and 1×1 convolution to obtain channel weights. And use global max pooling (GMP) and 7×7 convolution to obtain spatial weights The process is expressed as:

[0176] According to the feature map f 3 Generate channel weights

[0177]

[0178] Generate channel weights

[0179]

[0180] Among them, Conv 1×1 represents 1×1 convolution, σ represents activation function, GAP represents global average pooling, and GMP represents global maximum pooling;

[0181] Finally, the prompt parameter p is generated 3 :

[0182]

[0183] where ⊙ represents element-wise multiplication.

[0184] The generated cues share relevant representations and guide spatial reflective perception.

[0185] Step S3.3: Update prompt parameters

[0186] Although the prompt chain generated by the prompt chain generation module, i.e., the PCG module, has achieved overall reasoning, there is a lack of coupling between individual prompts, which may lead to the forgetting of key features. Inspired by dense connections (Huang et al., 2017), when j < 3, we fuse the first 3-j prompts together, and when j = 3, we keep the original prompts. However, fusing past prompts is inevitable and will introduce noise and redundancy. Therefore, the present invention refines them based on a gating mechanism and proposes a prompt updating (PU) module based on a gating mechanism.

[0187] When j=1, the present invention uses channel shuffling (Zhang et al., 2018) to process p 3 Then, the original features are fused and generated by dimensionality reduction projection. Specifically, it can be expressed as:

[0188] In this embodiment, if Figure 1 , Figure 4 As shown in (a),

[0189] In the prompt update module, generate updated prompt parameters

[0190]

[0191] Among them, Shuffle means channel shuffle.

[0192] Step S3.4: Generate feature map in the hint injection module And prompt parameters

[0193] The hints refined by the Prompt Updating (PU) module can guide the localization of the mirrored area. To this end, we use the Prompt Injection (PI) module to inject hints into the feature map and iterate the hints through interaction. To further make the hints aware of the difference between the image and the entity, the present invention introduces a prediction mask generated from the current level to decouple the mirrored area and the non-mirrored area.

[0194] In this embodiment, the detailed structure of the prompt injection module is as follows: Figure 5 As shown, specifically, according to (Zamir et al., 2022), the present invention uses three groups of 1×1 and 3×3 deep convolutions to encode local features and map to generate queries, keys, and values. Specifically:

[0195] In the prompt injection module, first according to the prompt parameters Generate feature map

[0196]

[0197] Among them, Conv 1×1 Represents 1×1 convolution, Conv 1×1 represents 3×3 convolution, and Split represents the division into three feature maps by channel, namely

[0198] In the same way, based on the feature map f 3 Generate feature map

[0199] Then the fusion prediction graph o 3 With feature map Get mirror feature

[0200]

[0201] Similarly, the correlation matrix Corr from feature map to prompt can be obtained p→f 、Corr f→p . Unlike previous work (Huang et al., 2023a) that modeled correlations between features from the spatial dimension, we consider it from the channel perspective for two reasons: 1) Channels better capture semantic information, which is more consistent with the original intent of implicit cues. 2) It is less computationally expensive. We observe that compared to spatial modeling, channel-based modeling requires at least half the GPU memory and reduces the computational cost while maintaining comparable performance, specifically:

[0202] Generate the correlation matrix Corr from the prompt to the feature map p→f :

[0203]

[0204] Where τ is a learnable scaling factor;

[0205] At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p :

[0206]

[0207] Finally, we get the feature map with the hint injected And the updated prompt parameters

[0208]

[0209]

[0210] in, Represents matrix multiplication.

[0211] Step S4: Decode and generate prediction graph o 2 And update the feature map required for decoding the next scale And prompt parameters

[0212] Step S4.1: Figure 1 As shown, in the aggregation prediction module, the feature map Aggregate with feature map S3 to obtain aggregate feature F 2 :

[0213]

[0214] Then, the aggregate feature F 2 Convolution is performed separately to obtain the prediction graph o2 And the feature map f 2 .

[0215] Step S4.2: Generate prompt parameter p 2

[0216] In this embodiment, if Figure 1 , Figure 3 As shown in (b), in the prompt chain generation module, the prompt parameters are first Perform 3×3 transposed convolution and upsample by 2 times to get the prompt parameters

[0217] According to the feature map f 2 Generate channel weights

[0218]

[0219] Generate channel weights

[0220]

[0221] Finally, the prompt parameter p is generated 2 :

[0222]

[0223] Step S4.3: Generate updated prompt parameters

[0224] In this embodiment, if Figure 1 , Figure 4 As shown in (b), in the prompt update module, i.e., the PU module, the prompt parameter Upsampling, aggregation and redistribution of channel features to obtain prompt parameters

[0225]

[0226]

[0227]

[0228] Among them, Γ 1 , Γ 2 is the convolution + GELU activation function with different parameters, α and β are learnable parameters.

[0229] The high-frequency part of the image contains detailed representations, such as edges and textures, while the low-frequency part highlights the semantic information. Therefore, the present invention further filters the noise in the frequency domain. Inspired by (Pan, Cai, and Zhuang, 2022), we can generate high-frequency features by subtracting the low-frequency features obtained using a mean filter. The process is:

[0230] Then, generate high-frequency cue parameters

[0231]

[0232] Among them, Γ 3 It is a convolution + GELU activation function with different parameters;

[0233] Then, frequency aggregation is performed to generate the prompt parameter p′ 3 :

[0234]

[0235] Among them, ζ and γ are learnable parameters;

[0236] Then, the prompt parameter p′ 3 With the prompt parameter p 2 Connect by channel and get the prompt parameter p′ 2 , generate hint denoising parameters

[0237]

[0238] Step S4.4: Generate feature map in the hint injection module And prompt parameters

[0239] In this embodiment, if Figure 5 As shown, the feature map is generated in the prompt injection module, i.e., the PI module And prompt parameters

[0240] In the prompt injection module, first according to the prompt parameters Generate feature map

[0241]

[0242] In the same way, based on the feature map f 2 Generate feature map

[0243] Then use the prediction graph o 2 To decouple feature maps Get feature map

[0244]

[0245] Then generate the correlation matrix Corr from the prompt to the feature map p→f :

[0246]

[0247] At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p :

[0248]

[0249] Finally, we get the feature map with the hint injected And the updated prompt parameters

[0250]

[0251]

[0252] Step S5: Decode and generate prediction graph o 1 And update the feature map required for decoding the next scale And prompt parameters

[0253] This step is basically the same as step S4, and includes the following specific steps:

[0254] Step S5.1: Figure 1 As shown, in the aggregation prediction module, the feature map Aggregate with feature map S2 to obtain aggregate feature F 1 :

[0255]

[0256] Then, the aggregate feature F 1 Convolution is performed separately to obtain the prediction graph o 1 And the feature map f 1 ;

[0257] Step S5.2: Generate prompt parameter p 1

[0258] In this embodiment, if Figure 1 , Figure 3 As shown in (c), in the prompt chain generation module, i.e., P, the prompt parameters are first Perform 3×3 transposed convolution and upsample by 2 times to get the prompt parameters

[0259] According to the feature map f 1 Generate channel weights

[0260]

[0261] Generate channel weights

[0262]

[0263] Finally, the prompt parameter p is generated 1 :

[0264]

[0265] Step S5.3: Generate updated prompt parameters

[0266] In this embodiment, if Figure 1 , Figure 4 As shown in (c), in the prompt update module, i.e., the PU module, the prompt parameter Upsampling, aggregation and redistribution of channel features to obtain prompt parameters

[0267]

[0268]

[0269]

[0270] Then, generate high-frequency cue parameters

[0271]

[0272] Then, frequency aggregation is performed to generate the prompt parameter p′ 2 :

[0273]

[0274] Then, the prompt parameter p′ 3 , p′ 2 With the prompt parameter p 1 Connect by channel and get the prompt parameter p′ 1 , generate hint denoising parameters

[0275]

[0276] Step S5.4: Generate feature map in the hint injection module And prompt parameters

[0277] In this embodiment, if Figure 5 As shown, in the prompt injection module, first according to the prompt parameter Generate feature map

[0278]

[0279] In the same way, based on the feature map f 2 Generate feature map

[0280] Then use the prediction graph o 1 To decouple feature maps Get feature map

[0281]

[0282] Then generate the correlation matrix Corr from the prompt to the feature map p→f :

[0283]

[0284] At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p :

[0285]

[0286] Finally, we get the feature map with the hint injected And the updated prompt parameters

[0287]

[0288]

[0289] Step S6: Figure 1 As shown, in the aggregation prediction module, i.e., the AP module, the feature map Aggregate with feature map S1 to obtain aggregate feature F 0 :

[0290]

[0291] Then the aggregate feature F 0 Convolution is performed separately to obtain the prediction graph o 0 And the feature map f 0 ;

[0292] Step S7: Training the generalized implicit reflection perception network based on visual cues

[0293] The pyramid visual transformer network, feature aggregation module, aggregation prediction module, prompt chain generation module, prompt update module and prompt injection module constitute a generalized implicit reflection perception network based on visual prompts. The labeled image I in the training data set is processed according to steps S1 to S6 to obtain the prediction image o j ,j=0,1,2,3, based on the prediction graph o j Calculate the binary cross entropy loss separately and weighted intersection-over-union loss Get the total loss L under full supervision fully :

[0294]

[0295] Or based on the prediction graph j Calculate partial cross entropy loss separately and smoothing loss Get the total loss L under weak supervision weak :

[0296]

[0297] According to the total loss L fully Or the total loss L weakl Training a generalized implicit reflection perception network based on visual cues.

[0298] Step S8: Mirror detection

[0299] The image to be detected is sent to the generalized implicit reflection perception network based on visual cues, and processed according to steps S1 to S6 to obtain a refined prediction map o 0 , complete the mirror detection.

[0300] Figure 6: This is a diagram of the existing mirror detection and general two-segmentation framework, (a) is a simple encoding-decoding framework represented by HetNet (He, Lin and Lau 2023), (b) is a dual-stream architecture represented by SATNet, (c) is an explicit visual cue architecture represented by EVP, and (d) is the IRP architecture of the present invention, where (a), (b), and (c) have been described in the background technology. In contrast, the IRP framework proposed in the present invention does not require explicit generation of visual cues or adapters. On the contrary, the present invention learns implicit visual cues in the decoding stage. Frequency and depth cues inevitably introduce noise and are not ideal. In contrast, the implicit cues learned by the present invention have higher-order semantics, effectively avoid noise interference and enhance the representation of mirror areas. In addition, the present invention can be generalized to different tasks, such as COD. Through implicit learning, we can obtain symmetry perception from any angle, while HetNet or SATNet only establish entity-image correlation within a fixed range by randomly rotating 90 degrees (or multiples thereof). As Figure 7 As shown in (a) and (b), our method significantly outperforms the state-of-the-art methods with extremely low parameters and effectively handles reflection interference, irregular occlusion and scale modeling.

[0301] experiment

[0302] Datasets: We conduct experiments on 11 datasets.

[0303] Mirror dataset: The MSD and PMD (Lin, Wang, and Lau, 2020) datasets include 3063 and 5096 training images, and 955 and 571 test images, respectively. For the Mirror-RGBD dataset (Mei et al. 2021), it includes 2000 training images and 1049 test images, all with depth maps. The VMD dataset (Lin, Tan, and Lau 2023) has 143 videos (7835 images) and 126 videos (7152 images) for training and testing.

[0304] Glass Dataset: There is GDD (Mei et al. 2020), which includes 2980 and 936 images for training and testing, respectively.

[0305] Camouflage scene datasets: CAMO (Le et al., 2019) has 1000 and 250 images for training and testing, respectively. COD10K (Fan et al., 2020) includes 3040 training images and 2026 test images. NC4K (Lv et al., 2021) includes 4121 images and is only used for testing.

[0306] Underwater scene datasets: MAS3K (Li et al., 2020a) RMAS (Fu et al., 2023) and UFO120 (Islam, Luo, and Sattar, 2020) include 1769 / 2514 / 1500 training images and 1141 / 500 / 120 test images respectively.

[0307] Implementation details: We implement the invention via PyTorch and conduct all experiments on NVIDIA A100. We then use a PVT network pre-trained on ImageNet as the backbone. For fair comparison, the input is resized to 384×384 for all datasets. For training, we use Adam W (Loshchilow and Hutter 2017) as the optimizer to update the model parameters for 200 epochs with a batch size of 40 images and an initial learning rate of le-4. For testing, we directly compute the resulting prediction maps without using any post-processing operations (e.g., CRF) to refine the prediction results.

[0308] Evaluation indicators: We use five evaluation indicators: mean absolute error (MAE), S-measure (S m ), average E-measure (E m ), weighted F-measure (Fan et al., 2017, 2018), Intersection over Union (IoU). For the last four evaluation indicators, the higher the better. It should be noted that all evaluation data are calculated based on the prediction results provided by the original paper or obtained by pre-training using the official code.

[0309] Comparison with existing methods

[0310] Quantitative comparison: We select different existing methods with different backbone networks (CNN or Transformer based) and different parameter settings to compare and confirm the advancedness of the IPR framework of the present invention. In particular, as shown in Table 1, for full supervision, we select five mirror detection methods: MirrorNet (Yang et al., 2019), PMDNet (Lin, Wang, and Lau 2020), HetNet, SATNet, and CSFwinformer (Xie et al. 2024). Three SOD methods: MENet (Wang et al. 2023a), SETR (Zheng et al. 2021) and VST (Liu et al. 2021) Three COD methods: FSPNet (Huang et al. 2023b), FPNet (Cong et al. 2023b) and VSCode, and two general segmentation methods: SAM, DualSAM (Zhang et al. 2024) and EVP.

[0311]

[0312] Table 1

[0313] Table 1 is a quantitative comparison on the MSD and PMD datasets. S, C, G, W, and M represent SOD, COD, general segmentation, weakly supervised mirror detection, and fully supervised mirror detection, respectively. The best performance is indicated in bold, and Ours is the present invention.

[0314] As can be seen from Table 1, the present invention surpasses various methods, especially EVP, exceeding 3.8%, 7.8%, 6.0%, 10.8% and 12.0% in the five evaluation indicators of the MSD dataset respectively, and exceeds CSFwinformer by an average of about 5%, which demonstrates the effect of implicit prompt learning.

[0315]

[0316] Table 2

[0317] Table 2 is a quantitative comparison on the Mirror-RGBD dataset. As shown in Table 2, we choose the RGB-DSOD method, namely PICRNet (Cong et al., 2023a), and the MD-specific method, namely PDNet (Mei et al., 2021). The present invention achieves better performance than PDNet without the assistance of a deep model. For the weakly supervised setting, the present invention also exceeds WSMD.

[0318]

[0319] Table 3

[0320] Table 3 is a comparison of model efficiency. As shown in Table 3, we compare the present invention with five mirror detection models in terms of parameters (M) and FLOPs (GMAC). The parameters and FLOPs of the present invention are 1 / 10 and 1 / 5 of those of CSFwinformer, respectively, which proves its efficiency.

[0321]

[0322] Table 4

[0323] Table 4 is a quantitative comparison on the VMD dataset.

[0324]

[0325] Table 5

[0326] Table 5 is a quantitative comparison on the GDD dataset.

[0327]

[0328] Table 6

[0329] Table 6 is a quantitative comparison in camouflage and underwater scenes.

[0330] As shown in Tables 4, 5, and 6, the existing methods for each task (such as VMDNet (Lin, Tan, and Lau 2023) and RFENet) are selected to verify the versatility of the present invention.

[0331] Qualitative comparison: We consider different scenarios and provide visual comparisons. Figure 8 As shown, the first two rows (a) and (b) show the presence or absence of corresponding entities, and the third and fourth rows (c) and (d) show regular and irregular occlusions. Our method establishes the connection between the entity and the image and the surrounding images. The last two rows (e) and (f) show scale changes and multiple targets. Our method (the present invention) can model globally and avoid missed reports and false reports.

[0332] Ablation studies

[0333]

[0334] Table 7

[0335] Table 7 shows the qualitative ablation of the proposed module. The first row is the baseline model, when the PI module is not used, we inject cues using the channel connection method.

[0336]

[0337] Table 8

[0338] Table 8 shows the quantitative ablation of the PI module, M i Indicates whether the non-mirror area mask is fused at level i.

[0339]

[0340] Table 9

[0341] Table 8 shows the effects of different numbers of prompts.

[0342] We validated the proposed modules (see Table 7 and Fig. 9 ), mask guidance (Table 8), and the effect of the number of hints (Table 9) on the MSD dataset.

[0343] Impact of PCG module: We added the PCG module to the baseline model and obtained 1.3%, 1.1% and 4.0% improvements on the four evaluation indicators. Fig. 9As shown in the second and third images, the occlusion problem is alleviated, but the module introduces some noise, which leads to false detection in the mirror area. In addition, the connection strategy is proved to be suboptimal because it lacks sufficient semantic and detailed interactions between cues and feature maps and introduces noise interference. We merge the PU and PI modules into the Baseline+PCG model to further improve the performance.

[0344] Impact of PU module: We add PU module based on Baseline+PCG model. Fig. 9 As shown in the fourth image, false detections are reduced and the occlusion effect is further reduced. This shows that the PU module effectively filters out noise and enhances the correlation between cues, generating a clean and informative chain.

[0345] Impact of PI module: By adding the PI module, we surpassed the Baseline+PCG model but did not reach the Baseline+PCG+PU model, indicating that the strong coupling cue (chain) is the most critical, while the injection strategy is supportive but also essential. Fig. 9 As shown in the fifth image, the detected mirror area is incomplete.

[0346] Specular Mask: At different stages of the PI module, we fuse feature maps and specular or non-specular region masks to enhance perceptual differences. The performance gradually improves with the increase of specular masks. However, the performance drops sharply when it is used in all stages. Therefore, perceiving only the specular region is not enough; non-specular regions are also necessary to build a complete context.

[0347] Number of prompts: When N = 0, there is no PU and PI module, which is the baseline model. As N increases, S m and FLOPs gradually improve. However, when N ≥ 4, S m and the rate of increase of FLOPs starts to decrease. Although we can achieve higher performance, it also introduces a significant computational burden. Therefore, N=3 is the optimal balance between performance and efficiency.

[0348] Failure scenario analysis and wider impacts

[0349] like Fig.10 As shown, our method (the present invention) performs relatively poorly when facing highly irregular occlusions. The present invention is not limited to mirror detection and can be used in various scenarios, such as camouflage.

[0350] in conclusion

[0351] We rethink the existing models for the mirror detection task and propose the IRP framework. Unlike previous methods that rely on explicit rotation or visual cues, our model allows for effective perception of the representation difference between the entity and the image, thereby achieving accurate and complete localization of the mirror area. Our framework outperforms existing methods on 11 benchmarks with lower computational complexity.

[0352] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A mirror image detection method based on generalized implicit reflection perception, characterized in that: The following steps are involved: (1) Generate a multi-scale feature map X i Use the Pyramid Vision Transformer (PVT) network as the backbone network to transform the input image I∈R 3×H×w Extracting multi-scale long-range dependency feature maps generates multi-scale feature maps where i∈{1,2,3,4}, R represents space, C i represents the number of channels of the i-th scale, H and W represent the height and width respectively; (2) Gradually aggregate adjacent features to generate feature map S i In the feature aggregation module, the multi-scale feature map X i With X i-1 , i∈{2,3,4} is aggregated to obtain the feature map S i : S i =Conv(Concat(UP↑(X i ),X i-1 )) The multi-scale feature map X4 is aggregated with the feature map S2 to obtain the feature map S1: S1=Conv(Concat(S2,Conv(X4))) Among them, UP↑ means upsampling, Concat means channel connection, and Conv means convolution; (3) Decoding generates prediction graph o 3 And update the feature map required for decoding the next scale And prompt parameters 3.1) In the aggregation prediction module, the feature map Conv(X4) after convolution of the multi-scale feature map X4 is first aggregated with the feature map S4 to obtain the aggregated feature F 3 : F 3 =Conv(Concat(Conv(Conv(X4)),Conv(S4))) Then, the aggregate feature F 3 Convolution is performed separately to obtain the prediction graph o 3 And the feature map f 3 ; 3.2) Generate prompt parameter p 3 In the prompt chain generation module, the prompt parameters are first randomly initialized According to the feature map f 3 Generate channel weights W c 3 =Conv 1×1 (σ(Conv 1×1 (GAP(f 3 )))) Generate channel weights W s 3 : W3 3 =Conv 7×7 (concat(GAP(f 3 ),GMP((f 3 )))) in, Conv 1×1 represents 1×1 convolution, σ represents activation function, GAP represents global average pooling, and GMP represents global maximum pooling; Finally, the prompt parameter p is generated 3 : Among them, ⊙ represents element-by-element multiplication; 3.3) In the prompt update module, generate updated prompt parameters Among them, Shuffle means channel shuffle; 3.4) Generate feature maps in the prompt injection module And prompt parameters In the prompt injection module, first according to the prompt parameters Generate feature map Among them, Conv 1×1 Represents 1×1 convolution, Conv 1×1 represents 3×3 convolution, and Split represents the division into three feature maps by channel, namely In the same way, based on the feature map f 3 Generate feature map Then the fusion prediction graph o 3 With feature map Get mirror feature Then generate the correlation matrix Corr from the prompt to the feature map p→f : Where τ is a learnable scaling factor; At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p : Finally, we get the feature map with the hint injected And the updated prompt parameters in, Represents matrix multiplication; (4) Decoding generates prediction graph o 2 And update the feature map required for decoding the next scale And prompt parameters 4.1) In the aggregation prediction module, the feature map Aggregate with feature map S3 to obtain aggregate feature F 2 : Then, the aggregate feature F 2 Convolution is performed separately to obtain the prediction graph o 2 And the feature map f 2 ; 4.2) Generate prompt parameter p 2 In the prompt chain generation module, the prompt parameters are first Perform 3×3 transposed convolution and upsample by 2 times to get the prompt parameters According to the feature map f 2 Generate channel weights W c 2 : W c 2 =Conv 1×1 (σ(Conv 1×1 (GAP(f 2 )))) Generate channel weights W s 2 : W s 2 =Conv 7×7 (concat(GAP(f 2 ),GMP((f 2 )))) Finally, the prompt parameter p is generated 2 : 4.3) Generate updated prompt parameters In the prompt update module, change the prompt parameter Upsampling, aggregation and redistribution of channel features to obtain prompt parameters in, Γ 1 , Γ 2 is the convolution + GELU activation function with different parameters, α and β are learnable parameters; Then, generate high-frequency cue parameters Among them, Γ 3 It is a convolution + GELU activation function with different parameters; Then, frequency aggregation is performed to generate the prompt parameter p ′3 : Among them, ζ and γ are learnable parameters; Then, the prompt parameter p ′3 With the prompt parameter p 2 Connect by channel and get the prompt parameter p ′2 , generate hint denoising parameters 4.4) Generate feature maps in the prompt injection module And prompt parameters In the prompt injection module, first according to the prompt parameters Generate feature map In the same way, based on the feature map f 2 Generate feature map Then use the prediction graph o 2 To decouple feature maps Get feature map Then generate the correlation matrix Corr from the prompt to the feature map p→f : At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p : Finally, we get the feature map with the hint injected And the updated prompt parameters (5) Decoding generates prediction graph o 1 And update the feature map required for decoding the next scale And prompt parameters 5.1) In the aggregation prediction module, the feature map Aggregate with feature map S2 to obtain aggregate feature F 1 : Then, the aggregate feature F 1 Convolution is performed separately to obtain the prediction graph o 1 And the feature map f 1 ; 5.2) Generate prompt parameter p 1 In the prompt chain generation module, the prompt parameters are first Perform 3×3 transposed convolution and upsample by 2 times to get the prompt parameters According to the feature map f 1 Generate channel weights W c 1 : W c 1 =Conv 1×1 (σ(Conv 1×1 (GAP(f 1 )))) Generate channel weights W s 1 : W s 2 =Conv 7×7 (concat(GAP(f 1 ),GMP((f 1 )))) Finally, the prompt parameter p is generated 1 : 5.3) Generate updated prompt parameters In the prompt update module, change the prompt parameter Upsampling, aggregation and redistribution of channel features to obtain prompt parameters Then, generate high-frequency cue parameters Then, frequency aggregation is performed to generate the prompt parameter p ′2 : Then, the prompt parameter p ′3 、p ′2 With the prompt parameter p 1 Connect by channel and get the prompt parameter p ′1 , generate hint denoising parameters 5.4) Generate feature maps in the prompt injection module And prompt parameters In the prompt injection module, first according to the prompt parameters Generate feature map In the same way, based on the feature map f 2 Generate feature map Then use the prediction graph o 1 To decouple feature maps Get feature map Then generate the correlation matrix Corr from the prompt to the feature map p→f : At the same time, we get the correlation matrix Corr from the feature map to the prompt f→p : Finally, we get the feature map with the hint injected And the updated prompt parameters (6) In the aggregation prediction module, the feature map Aggregate with feature map S1 to obtain aggregate feature F 0 : Then the aggregate feature F 0 Convolution is performed separately to obtain the prediction graph o 0 And the feature map f 0 ; (7) Training a generalized implicit reflection perception network based on visual cues The pyramid visual transformer network, feature aggregation module, aggregation prediction module, prompt chain generation module, prompt update module and prompt injection module constitute a generalized implicit reflection perception network based on visual prompts. The labeled image I in the training data set is processed according to steps (1) to (6) to obtain the prediction image o j ,j=0,1,2,3, based on the prediction graph o j Calculate the binary cross entropy loss separately and weighted intersection-over-union loss Get the total loss L under full supervision fully : Or based on the prediction graph j Calculate partial cross entropy loss separately and smoothing loss Get the total loss L under weak supervision wea : According to the total loss L fully Or the total loss L wea Training a generalized implicit reflection perception network based on visual cues; (8) Mirror detection The image to be detected is sent to the generalized implicit reflection perception network based on visual cues, and processed according to steps (1) to (6) to obtain a refined prediction map o 0 , complete the mirror detection.

Images