Retrieval method and device based on foreground and background segmentation mask, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a retrieval method, device, equipment and medium based on foreground and background segmentation masks.The retrieval method comprises the steps that multi-scale features are extracted from an input image, and the foreground and background segmentation masks are generated; performing spatial weighting by using the segmentation mask to obtain background weighting features; generating a background embedding vector through pooling and normalization, and storing the background embedding vector into a dynamic memory bank; negative samples are screened by adopting double thresholds according to the similarity with historical vectors; embedding a vector in combination with a positive sample generated by data enhancement to update a network; processing the query image to generate a query background embedding vector; and performing similarity matching with a dynamic memory library, and outputting a retrieval result list. According to the method, through combination of segmentation guidance and dynamic contrast learning, discriminative background features are obtained, negative samples are adaptively screened, collaborative optimization of segmentation and retrieval is realized, and the accuracy and robustness of cross-scene retrieval are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a retrieval method, apparatus, device, and storage medium based on foreground and background segmentation masks. Background Technology

[0002] In the field of computer vision, background segmentation and retrieval techniques have been widely applied in image editing, augmented reality, and video surveillance. However, traditional methods often rely on manually designed features (such as color histograms or texture descriptors) for background analysis. These methods have limited performance in complex scenes, especially lacking sufficient robustness when dealing with background diversity, lighting variations, and local interference. With the development of deep learning, segmentation models based on U-Net and Mask R-CNN have achieved significant improvements in foreground and background separation accuracy. However, these methods primarily focus on segmentation accuracy and lack the ability to discriminately model background features, making it difficult to generate stable and discriminative feature representations in subsequent retrieval tasks.

[0003] In the fintech sector, background segmentation and retrieval technologies can be applied to scenarios such as automated ID verification, transaction video surveillance, and remote identity verification. Existing methods mostly rely on global image features or fixed templates for matching. When similar background interference exists, this can easily lead to false positives or false negatives, reducing the system's reliability in high-security scenarios. Furthermore, traditional contrastive learning methods often employ randomly selected negative samples or statically set thresholds, which can cause the system to fall into two categories of problems: simple negative samples are insufficient to provide effective distinguishing signals, while excessively difficult negative samples can easily introduce optimization instability. This problem is particularly prominent in financial risk control systems. For example, in transaction scenario monitoring, complex backgrounds can lead to biases in the extraction of user behavior features, thus affecting the accuracy of anomaly detection.

[0004] In the healthcare field, background segmentation and retrieval technologies also have wide-ranging applications. For example, in medical image-assisted diagnosis, intraoperative navigation, and pathological slide analysis, accurate identification and modeling of background regions are crucial. Existing methods often focus only on the segmentation effect of foreground targets (such as lesion regions), neglecting the discriminative modeling of background features. Since background regions in medical images often contain noise or artifacts, the lack of effective background modeling methods leads to fuzzy embedded feature representations, thereby reducing the accuracy of retrieval and comparative analysis. Furthermore, existing negative sample screening mechanisms do not consider the hierarchical structure of background features, making it difficult to effectively exclude background regions highly similar to lesions in medical image retrieval, easily leading to misdiagnosis or uncertainty in assisted diagnosis. Summary of the Invention

[0005] The main objective of this invention is to provide a retrieval method, apparatus, device, and storage medium based on foreground and background segmentation masks, aiming to solve the technical problems of existing technologies lacking discriminative embedding modeling for background regions and adaptive negative sample screening mechanisms during training, resulting in a disconnect between segmentation and retrieval, unstable training, and poor cross-scene retrieval generalization ability.

[0006] To achieve the above objectives, the present invention provides a retrieval method based on foreground and background segmentation masks, comprising:

[0007] Acquire an input image and input the input image into a shared encoder to obtain multi-scale features;

[0008] The multi-scale features are input into the segmentation branch to generate foreground and background segmentation masks;

[0009] Based on the foreground and background segmentation mask, the multi-scale features are spatially weighted to obtain background weighted features;

[0010] Pooling and normalization operations are performed on the background weighted features to generate a background embedding vector, and the background embedding vector is added to the dynamic memory.

[0011] Based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory, the negative sample set is filtered by dynamically adjusted dual thresholds.

[0012] Based on the positive sample embedding vector generated by data augmentation and the negative sample set, update the shared encoder, segmentation branch and embedding branch;

[0013] The query image is processed using the updated shared encoder, segmentation branch, and embedding branch to generate a query background embedding vector.

[0014] The query background embedding vector is matched with the background embedding vector in the dynamic memory to obtain a list of search results.

[0015] Furthermore, to achieve the above objectives, the present invention provides a retrieval device based on foreground and background segmentation masks, comprising:

[0016] The shared encoder module is used to acquire the input image and input the input image into the shared encoder to obtain multi-scale features;

[0017] The segmentation branch module is used to input the multi-scale features into the segmentation branch to generate foreground and background segmentation masks;

[0018] The spatial weighting module is used to perform spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features;

[0019] The embedding branch module is used to perform pooling and normalization operations on the background weighted features to generate a background embedding vector, and add the background embedding vector to the dynamic memory.

[0020] The dynamic memory module is used to filter the negative sample set based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory through dynamically adjusted dual thresholds.

[0021] The negative sample filtering module is used to update the shared encoder, segmentation branch, and embedding branch based on the positive sample embedding vector generated by data augmentation and the negative sample set.

[0022] The query processing module is used to process the query image using the updated shared encoder, segmentation branch, and embedding branch to generate a query background embedding vector.

[0023] The similarity retrieval module is used to perform similarity matching between the query background embedding vector and the background embedding vector in the dynamic memory to obtain a list of retrieval results.

[0024] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a retrieval program based on foreground and background segmentation masks stored in the memory and executable on the processor, wherein when the retrieval program based on foreground and background segmentation masks is executed by the processor, it implements the steps of the retrieval method based on foreground and background segmentation masks as described above.

[0025] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a retrieval program based on foreground and background segmentation masks, wherein the retrieval program based on foreground and background segmentation masks, when executed by a processor, implements the steps of the retrieval method based on foreground and background segmentation masks as described above.

[0026] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a retrieval method, apparatus, device, and medium based on foreground and background segmentation masks, comprising: acquiring an input image and inputting it into a shared encoder to generate multi-scale features; inputting the multi-scale features into a segmentation branch to generate foreground and background segmentation masks; performing spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features; performing pooling and normalization operations on the background weighted features to generate a background embedding vector and adding it to a dynamic memory; filtering a negative sample set based on the similarity between the background embedding vector and historical background embedding vectors in the dynamic memory using dynamically adjusted dual thresholds; updating the shared encoder, segmentation branch, and embedding branch based on the positive sample embedding vectors generated by data augmentation and the negative sample set; processing the query image using the updated shared encoder, segmentation branch, and embedding branch to generate a query background embedding vector; and performing similarity matching between the query background embedding vector and the background embedding vectors in the dynamic memory to obtain a retrieval result list. This invention extracts multi-scale features through a shared encoder and generates foreground and background segmentation masks using a segmentation branch. Combined with spatial weighting, it obtains more discriminative background weighted features, ensuring the representational power of the background embedding vector. It stores and updates the background embedding vector through a dynamic memory and introduces a dual-threshold mechanism to dynamically filter the negative sample set, preventing overly difficult or easy samples from affecting model training. The model is updated by comparing positive and negative samples, enabling collaborative optimization of the shared encoder, segmentation branch, and embedding branch. During the inference phase, the updated model is used to process the query image and retrieve the background based on similarity, achieving integrated segmentation and retrieval processing. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0028] Figure 1 This is a schematic diagram of an application environment for a retrieval method based on foreground and background segmentation masks according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating an embodiment of the retrieval method based on foreground and background segmentation masks according to the present invention.

[0030] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the retrieval device based on foreground and background segmentation mask of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0032] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0033] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0034] The retrieval method based on foreground and background segmentation masks provided in this invention can be applied to applications such as... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can obtain the input image from the user terminal and input it into the shared encoder to generate multi-scale features; input the multi-scale features into the segmentation branch to generate foreground and background segmentation masks; perform spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features; perform pooling and normalization operations on the background weighted features to generate background embedding vectors and add them to the dynamic memory; based on the similarity between the background embedding vector and the historical background embedding vectors in the dynamic memory, filter the negative sample set through dynamically adjusted double thresholds; update the shared encoder, segmentation branch, and embedding branch based on the positive sample embedding vectors generated by data augmentation and the negative sample set; use the updated shared encoder, segmentation branch, and embedding branch to process the query image to generate the query background embedding vector; perform similarity matching between the query background embedding vector and the background embedding vectors in the dynamic memory to obtain the retrieval result list. This invention extracts multi-scale features through a shared encoder and generates foreground and background segmentation masks using a segmentation branch. Combined with spatial weighting, it obtains more discriminative background weighted features, ensuring the representational power of the background embedding vector. It stores and updates the background embedding vector through a dynamic memory and introduces a dual-threshold mechanism to dynamically filter the negative sample set, avoiding the impact of overly difficult or easy samples on model training. The model is updated by comparing positive and negative samples, enabling collaborative optimization of the shared encoder, segmentation branch, and embedding branch. During the inference phase, the updated model is used to process the query image and retrieve the background based on similarity, achieving integrated segmentation and retrieval. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0035] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the foreground and background segmentation mask retrieval method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0036] like Figure 2 As shown, the retrieval method based on foreground and background segmentation masks proposed in this invention includes the following steps:

[0037] S10, acquire the input image, input the input image into the shared encoder, and obtain multi-scale features;

[0038] In this embodiment, acquiring the input image is the starting point of the entire processing flow. The input image can come from various sources, such as still images captured by a digital camera, CT or MRI images captured by medical imaging equipment, real-time images captured by surveillance cameras in financial settings, or local image files uploaded by users. Input images typically have different resolutions, formats, and noise characteristics. Therefore, before proceeding to subsequent processing, the input image needs to be standardized, including size normalization, color space conversion, and pixel intensity normalization. These preprocessing operations ensure that the input data enters the subsequent feature extraction stage at a uniform scale and distribution, thereby reducing instability caused by input differences.

[0039] The process of inputting an image into a shared encoder refers to the hierarchical feature extraction of the input image using a uniformly designed deep neural network module. A shared encoder typically consists of multiple convolutional layers, activation functions, and normalization layers, with different layers capturing information at different granularities. Initial convolutional layers extract low-level texture and edge information, such as blood vessel contours in medical images or the line structures on the surface of banknotes in financial scenarios. As the network depth increases, intermediate layers gradually aggregate regional patterns, such as the local morphology of organs or text regions on banknotes. Deeper layers form semantic features, capable of identifying lesions in medical images or abnormal markings in financial images.

[0040] Obtaining multi-scale features is a key output of the shared encoder, encompassing feature representations at different resolutions and receptive fields. Multi-scale features, by fusing shallow spatial details and deep semantic information, simultaneously capture both local fine structure and global context. This is typically achieved by upsampling or downsampling feature maps from different levels before concatenating or weighted fusion, ensuring that the final feature vector contains both low-level information such as edges and textures, as well as complex structures and global patterns. This multi-scale representation is crucial for background segmentation and subsequent feature embedding generation, as the background often contains both large-scale scene structure and localized texture interference.

[0041] In practical implementation, different encoder architectures can be chosen. For example, a hierarchical encoder based on convolutional neural networks can be used, constructing multi-scale features through layer-by-layer convolution and pooling; an attention-based encoding structure can also be used, enabling the model to adaptively capture salient regions of the input image. To adapt to different application scenarios, the input image size normalization strategy can also be adjusted. For example, medical images typically maintain high resolution to preserve details, while financial document images can be downsampled to improve computational efficiency without affecting the recognition of key elements. When fusing multi-scale features, direct concatenation can be used, or a learnable weighting mechanism can be used to automatically adjust the contribution ratio of features at different scales. For scenarios with high computational resources, additional pyramid structures can be added to generate more refined multi-scale features, while in resource-constrained scenarios, the number of scale levels can be reduced to achieve real-time processing.

[0042] Example: In the healthcare field, the input image can be a patient's MRI sequence. Multi-scale features extracted through a shared encoder can simultaneously capture the fine-grained texture of tumor margins and the macroscopic layout of surrounding organs, thus supporting accurate lesion segmentation and background modeling.

[0043] In the fintech business, the input image can be a scanned copy of a document. Multi-scale features obtained through a shared encoder can simultaneously identify the text structure and overall layout of the document. This ensures accurate modeling of background interference areas while avoiding instability in recognition results due to local noise, thus achieving higher reliability in document verification and risk management.

[0044] This embodiment effectively overcomes the problem of complex and diverse input image sources by uniformly inputting the input images into a shared encoder to obtain multi-scale features, ensuring the stability of subsequent segmentation and embedding modeling. The construction of multi-scale features enables the model to simultaneously possess local detail perception and global scene understanding capabilities, thereby enhancing the discriminativeness of background feature representation and improving the accuracy and robustness of subsequent retrieval or segmentation.

[0045] S20, input the multi-scale features into the segmentation branch to generate foreground and background segmentation masks;

[0046] In this embodiment, inputting multi-scale features into the segmentation branch is a crucial step in the background processing flow. Multi-scale features contain spatial and semantic information from different levels, which are further analyzed through structured convolution and decoding operations after being input into the segmentation branch. The segmentation branch design typically includes two parts: an encoding path and a decoding path. The encoding path is responsible for extracting high-dimensional semantic features related to segmentation at a deeper level, while maintaining the integrity of the input multi-scale features. In implementation, stacked convolutional blocks can be used to extract local patterns, and batch normalization and non-linear activation functions can be used for stable training.

[0047] The decoding path restores high-dimensional semantic features to a spatial resolution close to the original image through progressive upsampling operations. This process utilizes skip connections to fuse shallow features from the encoding path with upsampled features from the decoding path. This compensates for the spatial resolution limitations of deep features, ensuring the generated segmentation mask possesses rich semantic information while preserving edges and details. The fused features are typically integrated through convolutional layers and then input into an activation function to generate foreground and background segmentation masks. The generation of foreground and background segmentation masks signifies that the image is divided into target regions and background regions, forming a foundation that can be directly used for subsequent spatial weighting processing.

[0048] In implementation, the segmentation branch can adopt a U-Net-based architecture, utilizing a symmetrical encoder-decoder path to balance contextual and spatial information; alternatively, an attention-based segmentation network can be used, allowing the model to selectively focus on key regions of the foreground and background. For high-resolution medical images, the number of upsampling layers in the decoding path can be increased to ensure the masking precision meets clinical requirements. In financial image processing, the number of network layers can be reduced or a lightweight convolutional structure can be used to improve inference speed and adapt to real-time processing needs. During the fusion process, a simple concatenation operation can be used, or dynamic weighting can be achieved through learnable weights to make the feature contributions of different levels more reasonable.

[0049] Example: In the healthcare field, inputting multi-scale features from a patient's CT images into the segmentation branch can generate accurate foreground and background segmentation masks. For example, in lung images, the foreground region can correspond to lesions or organs, while the background region corresponds to non-target tissues. This process allows subsequent feature modeling to focus on the background, thereby improving the accuracy of lesion analysis and retrieval.

[0050] In the fintech field, inputting multi-scale features of the bill image into the segmentation branch can generate foreground and background segmentation masks. The foreground contains key text or seals, while the background corresponds to paper texture and non-key information. This approach avoids interference from the bill's main text during background retrieval, ensuring a more stable recognition and comparison process.

[0051] This embodiment achieves effective partitioning of the input image by inputting multi-scale features into the segmentation branch and generating foreground and background segmentation masks. This allows subsequent processing to perform weighted modeling of the background region. This process avoids the boundary blurring problem caused by relying on only a single layer of features, ensuring the accuracy of foreground and background segmentation, thereby improving the reliability of background embedding modeling and retrieval.

[0052] S30, Based on the foreground and background segmentation mask, the multi-scale features are spatially weighted to obtain background weighted features;

[0053] In this embodiment, spatial weighting of multi-scale features based on foreground and background segmentation masks is a key step in combining mask information with feature representation. Its purpose is to enhance the representational ability of the background region while suppressing interference from the foreground region. The foreground and background segmentation mask is essentially a spatial weight matrix, where each pixel corresponds to a probability or confidence level, used to mark whether the location belongs to the background or foreground. In practice, the confidence values ​​of the background region can be directly extracted to construct the background weight matrix, making it numerically close to a continuous probability distribution.

[0054] Multi-scale features, comprising low-level spatial features and high-level semantic features obtained from different convolutional layers, exhibit heterogeneity. If directly used for modeling, they are easily distorted by foreground features. Therefore, an element-wise weighting operation using a background weight matrix is ​​necessary to amplify the portions of the multi-scale features corresponding to the background region while weakening the response of the foreground region. The resulting feature map, after weighting, has a background region enhancement effect, providing a more discriminative input for subsequent pooling and embedding generation.

[0055] In the weighted processing, further feature enhancement operations can be performed, such as secondary modeling of the background region through convolution or attention mechanisms, so that the features focus more on the unique texture and structural information of the background while maintaining the original semantic expression. This spatial weighting is not just a simple numerical multiplication, but combines the spatial correspondence between the mask and the feature map to ensure that the background features occupy a dominant position in the global representation.

[0056] In implementation, the background confidence matrix can be directly combined with multi-scale features using element-wise multiplication, which is the most computationally efficient method. Alternatively, a channel attention mechanism can be introduced to further distinguish the importance of different channels based on element-wise weighting, thereby enhancing the background features semantically as well. For high-resolution scenarios, pyramid pooling combined with masking operations can be used to ensure that multi-scale features are enhanced in different receptive fields.

[0057] In medical imaging, fine-grained background confidence matrices can be used to precisely suppress non-target tissues, highlighting environmental features around lesions. In financial document processing, the complexity of the background weight matrix can be reduced, retaining only binary background markers for rapid filtering of non-critical information areas, ensuring real-time performance. For hardware-constrained environments, computational resource consumption can be reduced by compressing the resolution of the weight matrix or using sparse representation.

[0058] Example: In the healthcare business, combining the segmentation mask of lung CT images with multi-scale features can highlight the background information of normal tissues and weaken the lesion area in spatial weighted processing. This makes the background embedding vector more reflective of the individual's basic structural differences during retrieval, thereby assisting in the background comparison analysis between different patients.

[0059] In the fintech business, combining segmentation masks of bill images with multi-scale features can suppress foreground elements such as bill text and seals in spatial weighted processing, while enhancing background features such as paper texture and environmental noise. This allows for better differentiation between real and counterfeit backgrounds when searching for similar bills, thereby improving the reliability of financial bill retrieval and anti-counterfeiting.

[0060] This embodiment utilizes spatial weighting of multi-scale features based on foreground and background segmentation masks to effectively suppress foreground interference and enhance the expressive power of background region features. This processing makes the generated background-weighted features more stable in subsequent embedding modeling processes, avoiding ambiguity caused by the mixing of background and foreground features, thereby improving the accuracy and robustness of subsequent retrieval and contrastive learning.

[0061] S40, perform pooling and normalization operations on the background weighted features to generate a background embedding vector, and add the background embedding vector to the dynamic memory.

[0062] In this embodiment, the goal of performing pooling and normalization operations on the background-weighted features is to compress the spatially large feature representation into a compact vector expression, thereby facilitating subsequent storage and similarity calculation. After spatial weighting, each location of the background-weighted features carries the feature response of the background region, but these features still maintain a two-dimensional distribution, making them unsuitable for direct use as embedding representations. Therefore, pooling operations are needed to perform global statistics on the entire feature map. Global average pooling can aggregate the feature intensity of each channel, outputting a channel-level average vector, so that the background features at different locations are statistically uniformly represented.

[0063] After pooling, normalization is required. Different channels of background features may differ significantly in their numerical range and distribution. Without normalization, subsequent similarity calculations will favor channels with larger numerical values. Normalization can be achieved using L2 norm standardization, projecting the entire pooling vector onto a unit sphere, thus generating a background embedding vector with consistent orientation and scale. This normalized vector ensures more stable comparison results of background features under different input conditions.

[0064] After the background embedding vectors are generated, they need to be stored in a dynamic memory. A dynamic memory is an updatable data storage structure that maintains a set of historical background features to provide a reference for subsequent contrastive learning or retrieval tasks. To prevent unlimited storage growth, dynamic memories typically employ a first-in, first-out (FIFO) strategy; when the storage capacity is reached, the oldest stored vectors are removed. Furthermore, existing vectors can be updated using moving averages or exponential moving averages, allowing the memory to dynamically adapt to the latest distribution while avoiding drastic oscillations.

[0065] In implementation, pooling features can be obtained through global average pooling, or global max pooling can be used to highlight extreme responses in the background features. To further improve background discrimination capabilities, weighted pooling can be combined, summing the results at different locations based on background confidence levels to more accurately capture significant background regions. For normalization, L2 normalization, batch normalization, or layer normalization strategies can be introduced to adapt to different hardware and task requirements.

[0066] Dynamic memory storage strategies can be adjusted according to different scenarios. During the training phase, a smaller capacity and faster update mechanism can improve model convergence efficiency; during the inference phase, a larger capacity and slower update mechanism can be used to maintain a long-term stable background feature library. In medical imaging, a hierarchical memory can be designed, storing background embedding vectors of different tissue types in partitions to improve the efficiency of retrieval for specific diseases. In financial instrument scenarios, a smaller capacity and a strict FIFO strategy can be set to ensure that the vectors in the memory always reflect the latest instrument background distribution.

[0067] Example: In the healthcare business, background region features in a patient's chest X-ray are pooled and normalized to generate a background embedding vector, which is then stored in a dynamic memory. This allows for the rapid identification of similar background structures among different patients during subsequent retrieval, thereby assisting doctors in discovering potential abnormal patterns in group data.

[0068] In the field of fintech business, generating background embedding vectors from the paper texture and printing environment information in scanned bills and storing them in a dynamic memory can identify the similarity of backgrounds of different bills during the retrieval process. This helps to detect duplicate printing, forgery, or background tampering, thereby improving the security and reliability of financial bill processing.

[0069] This embodiment achieves efficient transformation from high-dimensional dense features to compact embedding vectors by performing pooling and normalization operations on background weighted features and storing them in a dynamic memory, while ensuring the stability of cross-sample comparisons. The introduction of the dynamic memory allows the background embedding vectors to be continuously compared with historical background features, providing a dynamic reference for negative sample screening and retrieval tasks, thereby improving the discriminative power and generalization ability of the background representation.

[0070] S50, based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory, the negative sample set is filtered by dynamically adjusted dual thresholds;

[0071] In this embodiment, when calculating the similarity between the background embedding vector and the historical background embedding vectors in the dynamic memory, cosine similarity is typically used to measure the angular relationship between the two vectors. This method can eliminate the interference of amplitude differences on the comparison results and better reflect the consistency in feature directions. By calculating the similarity between the current background embedding vector and all historical background embedding vectors in the memory one by one, a set of similarity values ​​can be obtained.

[0072] To select suitable background embedding vectors as negative samples, upper and lower similarity thresholds need to be set. The lower threshold is used to remove historical features that differ too much from the current background embedding vector. Although these vectors are different, they are too simple and cannot provide effective contrastive learning signals. The upper threshold is used to remove historical features that are too close to the current background embedding vector. These vectors are close to positive samples and can easily cause confusion in model discrimination. Finally, historical background embedding vectors that fall between the upper and lower thresholds are selected to form the negative sample set.

[0073] The dual thresholds are not fixed but updated dynamically. As training progresses, the model's ability to discriminate background features gradually improves, and the original thresholds may no longer be suitable. Therefore, the threshold values ​​need to be gradually adjusted using moving averages or exponential moving averages. This dynamic adjustment ensures that the selected negative samples remain within a moderate difficulty range, thus providing effective learning signals without causing the model to fall into unstable optimization.

[0074] In practical implementation, different similarity calculation methods can be used. Cosine similarity is the most commonly used metric, but Euclidean distance or Mahalanobis distance can also be introduced as supplements to adapt to different feature distribution scenarios. Dynamic updates to the threshold can be achieved through simple linear decreasing or increasing methods, or through adaptive adjustment based on validation set performance feedback, to ensure that the screening results match the training objective.

[0075] When training on large-scale datasets, a random sampling strategy can be employed, extracting only a portion of historical embedding vectors from the dynamic memory to calculate similarity, thereby reducing computational overhead. In scenarios with high real-time requirements, similarity calculation can be parallelized and the filtering process can be completed on the GPU to ensure the overall system speed.

[0076] Example: In the healthcare field, similarity calculations are performed between the background embedding vectors of patient images and the background embedding vectors of historical cases. By applying a dual-threshold filtering method, a set of comparison samples with moderate difficulty can be obtained. This mechanism helps train models capable of distinguishing different patient background features and reduces the interference of background noise on lesion identification.

[0077] In the field of fintech business, comparing the background embedding vector of a bill with the background embedding vector of historical bills, and using dual thresholds to filter out a set of backgrounds that are neither different from the current bill nor too similar, can be used to improve the robustness of counterfeit bill detection, thereby improving the security of financial bill processing.

[0078] This embodiment effectively avoids the problem of negative samples being too easy or too difficult by using a negative sample set selected based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory, combined with dynamically adjusted dual thresholds. The selected negative sample set can always remain within an appropriate difficulty range, providing stable and valuable learning signals, and significantly improving the model's discriminative ability and generalization performance.

[0079] S60, based on the positive sample embedding vector generated by data augmentation and the negative sample set, update the shared encoder, segmentation branch and embedding branch;

[0080] In this embodiment, the positive sample embedding vector generated based on data augmentation is compared with the negative sample set to form an effective optimization signal during training. The generation of the positive sample embedding vector relies on applying random perturbations to the input image, such as rotation, translation, scaling, color jitter, or Gaussian noise. These augmentation operations ensure that different versions of the same image maintain semantic consistency, thus being mapped to neighboring regions in the embedding space. The augmented image is input into a shared encoder to extract multi-scale features, which are then processed through a segmentation branch to obtain a segmentation mask. These multi-scale features are then combined with the mask and fed into the embedding branch, ultimately yielding the positive sample embedding vector.

[0081] Building upon this, the negative sample set originates from the historical background embedding vector set selected in the previous step using a dual-threshold mechanism. These negative samples are neither so different from the current positive samples that they lose their discriminative significance, nor so similar as to confuse the model, thus forming a stable training control. By calculating the similarity between the positive sample embedding vectors and the negative sample set, and combining this with a temperature coefficient to scale the contrast loss, the boundary distinguishing positive and negative samples in the embedding space can be strengthened.

[0082] Meanwhile, in addition to the contrastive loss, segmentation loss and cross-entropy loss also need to be calculated. Segmentation loss is based on the difference between the foreground / background segmentation mask and the supervision mask, while cross-entropy loss reflects the deviation between the segmentation classification result and the true label. These three losses are weighted and combined into a hybrid loss value. Finally, the hybrid loss is applied to the parameter updates of the shared encoder, segmentation branch, and embedding branch through the backpropagation process, enabling the model to improve the discriminative power of the background embedding vector while maintaining segmentation accuracy.

[0083] In practice, data augmentation methods can be chosen in various ways. Geometric transformations can be used, such as random rotations of 10 to 30 degrees or scaling to 80% to 120% of the original size; illumination changes can be used, such as randomly adjusting brightness and contrast; and noise augmentation can be introduced, such as superimposing Gaussian or salt-and-pepper noise on pixel values. Different augmentation strategies will affect the stability of the positive sample embedding vector and can be flexibly adjusted according to the specific application scenario.

[0084] The contrastive loss can be calculated using the standard InfoNCE loss or an angle-constrained loss function to improve the uniformity of the feature distribution. The temperature coefficient can be set to a fixed value, such as 0.07, or it can be gradually adjusted during training to balance the similarity distribution between samples. The weight ratio of segmentation loss to contrastive loss in the hybrid loss can also be adjusted. For example, in the early stages, the segmentation loss can have a higher weight to ensure mask quality, while in the later stages, the weight of contrastive loss can be gradually increased to optimize the embedding space.

[0085] In a distributed training environment, a cross-GPU negative sample sharing mechanism can be used to merge negative sample sets from different computing nodes, thereby improving the diversity of negative samples. In lightweight application scenarios, the complexity of augmentation operations can be reduced, retaining only a few key transformations to lower computational costs.

[0086] Example: In the healthcare field, data augmentation can be performed on medical images of the same patient, such as adding slight rotation and noise to CT slices, to generate positive sample embedding vectors. These vectors are then compared with background embedding vectors of other patients selected from a dynamic memory database, thereby training a model that can effectively distinguish patient background differences. This mechanism helps reduce the interference of background differences on lesion detection.

[0087] In the fintech field, random augmentation can be performed on the same invoice image, such as simulating changes in shooting angle or lighting, to obtain positive sample embedding vectors. These vectors are then compared with background embedding vectors of different invoices selected from a memory database to optimize invoice retrieval or counterfeit detection systems. This improves the system's accuracy and robustness in situations with complex and variable invoice backgrounds.

[0088] This embodiment generates positive sample embedding vectors through data augmentation and trains them using a dynamically selected set of negative samples. This effectively improves the discriminative power of the embedding space while maintaining segmentation accuracy. Positive sample augmentation maintains feature consistency, negative sample selection ensures moderate learning difficulty, and the introduction of a temperature coefficient improves the gradient distribution of the contrastive loss. Ultimately, this enables the shared encoder, segmentation branch, and embedding branch to optimize collaboratively, thereby enhancing the robustness and generalization of the overall system in complex scenarios.

[0089] S70 uses the updated shared encoder, segmentation branch, and embedding branch to process the query image and generate a query background embedding vector.

[0090] In this embodiment, the query image refers to the target image entering the system during the retrieval phase. Its source can be real-time camera capture, offline image library, push from business system interfaces, or cached data after unified preprocessing. After the query image enters the updated shared encoder, it first undergoes size alignment and pixel scale normalization processing according to the input specifications maintained during the training phase. This ensures that the channel order, dynamic range, and color space are consistent with those during training, avoiding representational bias caused by distribution drift. The updated shared encoder refers to a feature extraction network that has been trained and has its parameters frozen or semi-frozen. Its structure is consistent with the training phase, including initial convolutional units and multi-layer feature extraction stages, outputting multi-scale query features. Multi-scale refers to simultaneously providing feature map stacks with different spatial resolutions and receptive fields, such as higher-resolution texture-sensitive maps and lower-resolution semantic-sensitive maps, simultaneously preserving boundary details and regional semantics. This design is derived from the receptive field accumulation law of hierarchical convolutional networks and can be implemented through residual networks, densely connected networks, or backbones with dilated convolutions. The number of channels and scale levels remain consistent with those during training, avoiding scale mismatch.

[0091] The segmentation branch receives multi-scale features from the query and generates foreground and background segmentation masks. The segmentation branch employs a symmetrical structure of encoding and decoding paths. The encoding path continuously compresses spatial dimensions and accumulates semantics, while the decoding path progressively upsamples to restore resolution. Skip connections fuse shallow details from the encoding path with the upsampled features from the decoding path, ultimately outputting binary or probabilistic foreground and background masks. The probabilistic form expresses the confidence of the foreground and background within a numerical range, while the binary form obtains a clear outline through thresholding. The threshold can be fixed or adaptively estimated based on the mask histogram. This mask originates from the forward inference of the segmentation branch and is consistent with the supervised definition during training, ensuring that the mask semantics are aligned with the background features.

[0092] The query multi-scale features are coupled with the query foreground and background segmentation mask to generate weighted features for the query background. The weighting process uses the background confidence in the mask as spatial weights, multiplying the features at each scale pixel-by-pixel to amplify the representation of the background region and suppress foreground leakage. Weights can be directly derived from background probabilities, or smoothed using a monotonic mapping function to improve numerical stability. Consistency constraints can also be introduced across scales to ensure compatibility between the weights of high-resolution and low-resolution channels after downsampling or upsampling, reducing cross-scale noise. To reduce residual foreground contamination caused by jagged edges, low-amplitude edge smoothing or morphological thinning can be applied to the weight map to ensure that the dominant energy distribution of the background region is not altered.

[0093] The embedding branch performs pooling and normalization on the weighted features of the query background to generate a query background embedding vector. Pooling uses global averaging or weighted averaging, with the background weights as the spatial factor of the pooling kernel, compressing the spatial dimension into channel statistics. To avoid a small number of high-response pixels dominating the statistics, truncated averaging or quantile constraints can be introduced. The resulting channel vectors are scaled by linear or lightweight nonlinear projection and then L2 normalized, projecting the vectors onto a unit hypersphere to facilitate cosine similarity matching with the background embedding vectors in the dynamic memory. The vector dimension is consistent with that in the training phase to ensure comparability with the memory. To adapt to different hardware environments, half-precision or quantized tensors can be enabled before normalization to accelerate inference, while calibration maintains angular similarity unaffected by quantization offset.

[0094] The updated shared encoder, segmentation branch, and embedding branch maintain end-to-end forward consistency. The query image is processed by the shared encoder to obtain multi-scale query features, then by the segmentation branch to obtain foreground and background segmentation masks, followed by spatial weighting to obtain weighted background features, and finally by the embedding branch to output the background embedding vector. The entire process does not involve parameter updates and employs statistical and normalization strategies based on inference patterns. Batch normalization and other units use the accumulated mean and variance from the training period to avoid fluctuations introduced by online statistics. To improve throughput, multiple query images can be grouped into small batches for pipeline entry without altering the normalization properties of individual vectors. To adapt to ultra-high resolution images, overlapping slicing and sliding window aggregation can be used. Slice-level embeddings are aggregated into whole-image embeddings using area or confidence weighting, ensuring that the vector scale is consistent with the memory.

[0095] In resource-rich offline retrieval scenarios, multi-scale inference can be enabled. The query image is forward-engineered at two to three scaling scales to obtain multiple sets of query background embedding vectors. These are then weighted and fused to obtain the final vector, with weights derived from the mask quality evaluation scores at each scale. Alternatively, single-scale inference can be used on mobile or edge devices, with the number of backbone channels pruned. Grouped convolutions or depthwise separable convolutions can be used to reduce latency, while L2 normalization is preserved in the embedding branch to ensure compatibility with the angle metrics in the memory. Lightweight attention units can also be added to the segmentation branch. The attention weights are multiplied by the background confidence before participating in decoding fusion, thereby improving mask boundary alignment without significantly increasing computation.

[0096] For query images with extremely low contrast or drastic lighting fluctuations, a test-time adaptive normalization strategy can be introduced. Contrast stretching and color channel white balance correction are performed before entering the shared encoder. The correction parameters are obtained from image histogram statistics, resulting in a pixel distribution that more closely resembles the training distribution. In scenes sensitive to fine-grained background differences, a frequency domain filter channel can be added before the embedding branch to preserve mid-to-low frequency texture components and suppress high-frequency foreground edge interference. The filter kernel parameters are learned during training, and only forward convolution is performed during inference. For ultra-wide images or images with significant differences between long and short sides, a bidirectional pyramid input strategy can be adopted. Scale pyramids are constructed along both the long and short sides, and then aggregated in the embedding space after parallel inference by the shared encoder, improving adaptability to stretching distortion.

[0097] To ensure cross-device consistency, an inference-period calibration table can be introduced to record the quantization scale and zero point under different devices. Inverse quantization is performed before L2 normalization to ensure that angular similarity calculation is unaffected by device differences. To improve real-time performance, the forward graphs of the segmentation and embedding branches can be merged into a computational graph subgraph. Compiler kernel fusion can be enabled, fusing adjacent operators such as upsampling and pointwise convolution, element-wise multiplication and normalization into a single-core execution, reducing memory read / write overhead. For batch processing scenarios, a hybrid batch strategy can be used to allocate query images of the same resolution to the same batch, avoiding kernel scheduling overhead caused by dynamic shapes, while ensuring that the order of the output embedding vectors in each batch is aligned with the input, facilitating matching with the memory bank.

[0098] In applications with high security requirements, reversible noise protection can be applied to the query background embedding vector. This involves using zero-mean, small-variance Gaussian perturbations or orthogonal subspace perturbations to maintain the ranking within the cosine similarity framework, while simultaneously reducing the risk of plaintext vector theft. To improve cross-domain generalization, lightweight domain adaptation can be enabled. This involves linearly mapping the statistical distribution of the query image onto the input side, with mapping parameters derived from self-supervised estimations of a small number of unlabeled samples, without altering the shared encoder and branch parameters. For scenarios with mirror or rotation uncertainties, isovariant ensembles can be performed during inference. This involves horizontally flipping or performing small-angle rotations on the query image, and finally averaging or taking the principal component direction in the embedding space as the final vector, enhancing stability against pose changes.

[0099] Example Description: In the healthcare business domain, the input is a query set of endoscopic images. The updated shared encoder outputs query multi-scale features. The segmentation branch generates query foreground and background segmentation masks to mask instrument and mucosal highlights. Spatial weighting emphasizes cavity wall texture and blood vessel orientation. The embedding branch generates query background embedding vectors, which are used to retrieve case images with similar background conditions from the historical database. This helps to locate reference samples with consistent environments and reduce the interference of background differences on subsequent lesion comparisons.

[0100] In the fintech business, the input is a query set of images of bills or contracts. The updated shared encoder outputs multi-scale features of the query. The segmentation branch generates query foreground and background segmentation masks to isolate text and seals. Spatial weighting highlights paper fiber texture and printing background. The embedding branch generates query background embedding vectors, which are used to retrieve the real template with the closest background texture in the memory. This supports subsequent counterfeit identification and batch archiving retrieval, and reduces the impact of foreground text difference comparison on the results.

[0101] This embodiment processes the query image using an updated shared encoder, segmentation branch, and embedding branch consistency, resulting in a query background embedding vector that is in the same domain and has the same metric as the background representation constructed during the training phase. The segmentation mask spatially filters out foreground interference, spatial weighting suppresses foreground leakage in terms of features, and pooling and L2 normalization statistically eliminate scale and brightness differences. The combination of these three elements makes the vector more sensitive to background texture and layout, and less sensitive to foreground shape and color.

[0102] S80, perform similarity matching between the query background embedding vector and the background embedding vector in the dynamic memory to obtain a list of search results.

[0103] In this embodiment, the query background embedding vector and the background embedding vector in the dynamic memory are located in the same unit hypersphere space, and the metric is based on cosine similarity. To ensure comparability, the query background embedding vector has undergone L2 normalization during the generation stage, and entries within the dynamic memory also maintain the unit norm during addition and momentum maintenance. Similarity calculation is implemented using vector dot products, and the value is equivalent to cosine similarity. To reduce numerical instability caused by batch processing, a light temperature scaling can be performed before calculation, so that the dynamic range of the similarity distribution matches the subsequent sorting and thresholding strategies. The dynamic memory maintains the background embedding vector set in a first-in-first-out queue and includes metadata such as original image identifier, timestamp, background confidence, and source domain label. To improve retrieval efficiency, the vector set is built with an inverted index or near nearest neighbor index structure to accommodate millions of entries; the index can be based on hierarchical small-world graphs, product quantization, or inverted multi-clustering, and the index parameters are determined through cross-validation in the offline stage.

[0104] Similarity matching is implemented in a two-stage process. The first stage performs a coarse search in the index space, returning a candidate set and its approximate similarity. The second stage performs a fine search on the candidate set in the original vector space, calculating high-precision cosine similarity and performing stable ranking. To enhance robustness, quantile truncation and deduplication rules are introduced: for multiple embeddings of the same background image, only the one with the highest similarity is retained; diversity suppression is applied to candidates with overly concentrated source domains or shooting conditions to avoid results concentrated in a single scene. After ranking, a ranking sequence is formed, and grouping and rearrangement are performed when necessary, such as fine-tuning the order based on background confidence or time decay, so that recently added samples with high confidence are given appropriate priority. The search result list is represented by structured entries, which at least include background image identifiers, similarity scores, optional matching evidence (such as key area heatmap reference indexes), and link pointers available for subsequent business.

[0105] To adapt to different computing power environments, the matching process supports batch parallelism: multiple query background embedding vectors are combined into a matrix, and matrix multiplication is performed with the index cluster center or candidate set to obtain a similarity matrix, completing a large number of dot products in a single GPU core. For latency-sensitive online services, the index can be preheated to high-bandwidth GPU memory, and streaming Top-K operators can be enabled to avoid the time overhead of full sorting. To suppress false hits caused by noisy queries, the similarity distribution can be calibrated, mapping the original scores to a confidence space, and candidates below the confidence threshold are not included in the search results list. If cross-domain consistency is required, domain alignment weights can be introduced during matching, increasing the weight of background embedding vectors consistent with the query domain, and correcting cross-domain entries according to a learnable mapping to ensure that the ranking is not dominated by domain bias.

[0106] In large-scale retrieval environments, sharded indexing and gated routing can be employed. First, the dynamic memory is divided into several sub-databases based on the cluster centers of the background embedding vectors. When querying background embedding vectors, they are first matched against the centers of these sub-databases, and the most similar sub-databases are selected for secondary retrieval, reducing the traversal scope while maintaining recall. Alternatively, a compression-reconstruction strategy can be used. Vectors within the database are compressed through product quantization for coarse retrieval, and the original full-precision vectors are loaded onto the Top-M candidates for re-ranking, balancing speed and accuracy. Furthermore, a diversity regularization re-ranking algorithm can be introduced, applying redundancy reduction constraints based on background category labels, shooting angle, and texture spectral characteristics among candidates with similar similarity, thereby outputting a retrieval result list covering a wider range of background morphologies.

[0107] On resource-constrained devices, lightweight indexing and fixed-point inference can be enabled. Background embedding vectors are stored with an 8-bit or lower bit width, and the index uses compact hash buckets and locality-sensitive hash nearest neighbor filtering to achieve millisecond-level matching with constant-level memory. To offset quantization errors, mini-batch dequantization and high-precision rearrangement are performed on the final Top-K set to ensure the stability of the top results. In cross-regional deployments, hierarchical caching can be used to cache frequently hit background embedding vectors on the nearest node. When querying background embedding vectors, matching is first performed against the local cache, and only if no match is found is the global index retrieved from the source, significantly reducing cross-datacenter latency.

[0108] In security and compliance scenarios, reversible perturbation and access auditing can be added. The background embedding vectors in the dynamic memory are subjected to an orthogonal perturbation matrix during insertion, and the perturbation parameters are stored only within controlled hardware. During the matching phase, anti-perturbation and similarity calculations are performed within a secure execution environment. When the retrieval results list is returned, only necessary metadata and auditable logs are output, meeting privacy and traceability requirements. For highly precise retrieval with extremely high alignment requirements, local geometric consistency verification can be added: a background saliency heatmap is generated on the top-ranked candidate background images, and a region consistency comparison is performed with the background mask of the query image. Candidates that fail the comparison have their scores reduced or are removed, thereby reducing accidental high-similarity mismatches.

[0109] Example Explanation: In the healthcare business domain, the query background embedding vector is derived from the background representation of endoscopic images. A dynamic memory stores background embedding vectors from previous examinations and case indexes. Similarity matching first performs a coarse search on the sharded index, followed by a fine search and region consistency verification on the candidate set. The search results list returns the most similar cavity environment images and examination times, allowing doctors to quickly compare historical records under the same background conditions, reducing the interference of background differences on lesion comparison.

[0110] In the fintech business, the query background embedding vectors are derived from the paper texture and background structure of the scanned images of the invoices. The dynamic memory contains background embedding vectors from multiple batches of paper and printing batches. Similarity matching quickly locates similar batches on the quantitative index, and then the top-ranked samples are obtained by full-precision re-ranking. The search results list outputs the corresponding template and batch information, which assists in the verification of the authenticity of invoices and the consistency check of batch archiving, and reduces the impact of foreground text differences on search stability.

[0111] This embodiment achieves stable comparability between the query background embedding vector and the background embedding vector in the dynamic memory under angular metric by similarity matching in the unit hyperspherical vector space and a two-stage coarse-to-fine retrieval process. The combination of indexing and reordering significantly reduces computational cost while maintaining high accuracy. Diversity suppression and confidence calibration reduce result clustering and noise false hits. The resulting retrieval result list strikes a balance between similarity accuracy, coverage, and latency, maintaining consistent ranking quality across different computing power and cross-domain environments.

[0112] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a retrieval method, apparatus, device, and medium based on foreground and background segmentation masks, comprising: acquiring an input image and inputting it into a shared encoder to generate multi-scale features; inputting the multi-scale features into a segmentation branch to generate foreground and background segmentation masks; performing spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features; performing pooling and normalization operations on the background weighted features to generate a background embedding vector and adding it to a dynamic memory; filtering a negative sample set based on the similarity between the background embedding vector and historical background embedding vectors in the dynamic memory using dynamically adjusted dual thresholds; updating the shared encoder, segmentation branch, and embedding branch based on the positive sample embedding vectors generated by data augmentation and the negative sample set; processing the query image using the updated shared encoder, segmentation branch, and embedding branch to generate a query background embedding vector; and performing similarity matching between the query background embedding vector and the background embedding vectors in the dynamic memory to obtain a retrieval result list. This invention extracts multi-scale features through a shared encoder and generates foreground and background segmentation masks using a segmentation branch. Combined with spatial weighting, it obtains more discriminative background weighted features, ensuring the representational power of the background embedding vector. It stores and updates the background embedding vector through a dynamic memory and introduces a dual-threshold mechanism to dynamically filter the negative sample set, preventing overly difficult or easy samples from affecting model training. The model is updated by comparing positive and negative samples, enabling collaborative optimization of the shared encoder, segmentation branch, and embedding branch. During the inference phase, the updated model is used to process the query image and retrieve the background based on similarity, achieving integrated segmentation and retrieval processing.

[0113] In one embodiment, step S10 includes:

[0114] S101, acquire the input image, perform size standardization processing on the input image, and generate a standardized input image;

[0115] S102, the standardized input image is input into the initial convolutional layer of the shared encoder to extract primary texture features;

[0116] S103, the primary texture features are input into the first feature extraction stage of the shared encoder to generate a first-scale feature map;

[0117] S104, input the first scale feature map into the second feature extraction stage of the shared encoder to generate the second scale feature map;

[0118] S105, input the second scale feature map into the third feature extraction stage of the shared encoder to generate the third scale feature map;

[0119] S106, the primary texture features, the first-scale feature map, the second-scale feature map and the third-scale feature map are fused to form a multi-scale feature.

[0120] In this embodiment, the input image refers to the original image from the acquisition terminal or dataset. The format can be RGB or grayscale, the bit depth can be eight bits or higher, and the color space can be sRGB or the commonly used DICOM color space. To eliminate feature scale drift caused by differences in resolution and aspect ratio, size normalization processes perform both geometric and numerical normalization on the input image. Geometric normalization aligns the long or short side to a preset scale, maintaining the aspect ratio without distortion. Anti-aliasing interpolation is used, and zeros or mean padding is applied to the edges. This process ensures a stable mapping between the receptive field of subsequent convolutions and the physical size of pixels. Numerical normalization maps pixel values ​​to a fixed dynamic range and subtracts the mean and divides the variance by the channel. Before entering the subsequent network, the grayscale image is copied to three channels or increased in dimensionality through a one-by-one convolution layer to avoid weight sharing failure due to inconsistent channel numbers. After these processes, a normalized input image is obtained, possessing predictable spatial size, channel arrangement, and statistical distribution, which is beneficial for batch parallelism and hardware acceleration.

[0121] The initial convolutional layer receives a normalized input image and extracts primary texture features. The convolutional kernel spatially covers multiple adjacent pixels to encode edges, corners, and fine-grained textures. Stride and padding ensure that the output resolution matches the desired downsampling ratio. To improve numerical stability, normalization and non-linear activation can be applied after the initial convolution to constrain feature distribution and enhance discriminativity. If the input resolution is high, the initial convolution can be combined with lightweight pooling to achieve limited anti-aliasing downsampling, avoiding aliasing effects in the first feature extraction stage. The primary texture features cover multiple sets of filter responses along the channel dimension, providing directionality and frequency band information for subsequent stages.

[0122] The first feature extraction stage receives primary texture features and generates a first-scale feature map. Internally, it consists of several sequentially generated convolutional blocks. The kernel size and stride configuration create appropriate downsampling or maintain resolution. Residual connections or similar mechanisms ensure gradient propagation and training stability. This stage expands the effective receptive field while preserving a relatively fine spatial grid, suitable for representing edge continuity and texture repetition patterns. If the task domain contains noise or illumination variations, a convolutional configuration with strong bandpass characteristics can be used in this stage to suppress slowly changing components, thereby improving sensitivity to material boundaries.

[0123] The second feature extraction stage overlays deeper convolutional computations on top of the first-scale feature map to generate a second-scale feature map. To expand semantic coverage while controlling spatial degradation, two paths can be used: stride downsampling or dilated convolution. Stride downsampling improves abstraction and reduces computation, while dilated convolution maintains grid density and increases the receptive field. One path can be chosen based on data characteristics, or they can be parallelized in the same stage and then fused. The output of this stage encodes stronger contextual relationships in the channel dimension, such as the material consistency and layout trends of large background regions.

[0124] The third feature extraction stage continues to stack convolutional blocks based on the second-scale feature maps to generate a third-scale feature map. This stage focuses on acquiring global structural information within a larger receptive field, while suppressing discriminative loss caused by excessive smoothing through normalization and nonlinearity. If the first two stages have already produced a large downsampling ratio, convolutions with increasing dilatancy can be used here to avoid further loss of spatial resolution. To avoid the impact of channel amplitude differences at different scales on subsequent fusion, channel recalibration operations can be inserted at the end of the stage to perform bounded scaling of the response intensity, making the numerical ranges of features at each scale comparable.

[0125] The fusion operation aligns the primary texture features, first-scale feature maps, second-scale feature maps, and third-scale feature maps to a unified spatial grid and channel layout before aggregation. Spatial alignment can be achieved through upsampling or downsampling, and the interpolation method uses a smooth kernel to reduce ringing. Aggregation can be achieved through summation, concatenation with a one-to-one convolution, or weighted summation with learnable weights. Summation enables rapid information convergence, concatenation combined with a one-to-one convolution performs channel compression while preserving the independence of each scale, and the summation of learnable weights normalizes the weights of the four scales through soft maximization to form an adaptive ratio. To reduce the adverse effects of inter-scale phase deviation on aggregation, a lightweight alignment convolution can be added before alignment to compensate for the spectral loss caused by interpolation. The fusion output is a multi-scale feature, containing both the high-frequency local details carried by the primary texture features and the low-frequency global layout carried by the third-scale feature map, while retaining structural cues at intermediate scales. This representation remains differentiable and numerically stable in both spatial and channel dimensions, facilitating direct consumption by subsequent branches without requiring complex reorganization.

[0126] Throughout the process, the mapping between the input image and the standardized input image ensures stable pixel statistics entering the network; the initial convolutional layer guarantees effective frequency band decomposition of primary texture features; the three feature extraction stages expand the receptive field layer by layer and gradually abstract; the fusion stage completes cross-scale consistency modeling. Each named entity is explicitly generated and used in the preceding and following links, forming a coherent computation graph and avoiding dangling intermediate quantities or redundant computations. At the implementation level, tensor shapes can adopt a memory layout of batch size multiplied by channel number multiplied by height multiplied by width, and convolution and interpolation are performed with efficient operators; at the numerical level, the outputs of each stage are constrained by a unified normalization strategy to reduce the proportional imbalance during fusion; at the engineering level, spatial alignment and fusion can both be implemented using native operators of the automatic differential framework, supporting mixed precision and parallel pipelines, improving throughput and energy efficiency.

[0127] This embodiment stabilizes the spatial and statistical characteristics of the input through size normalization. The initial convolutional layer extracts primary texture features to capture high-frequency edges and fine-grained textures. The three feature extraction stages progressively expand the receptive field while preserving multi-granular structural cues. Fusion adaptively aggregates information from different scales onto a unified grid, thus ensuring that multi-scale features possess both local separability and global consistency. Subsequent segmentation branches obtain clearer boundaries and regional contexts on this representation, and the background representations extracted by the spatial weighting and embedding branches have lower noise and stronger discriminative power.

[0128] In one embodiment, step S20 above includes:

[0129] S201, the multi-scale features are input into the encoding path of the segmentation branch, and shallow features and deep semantic features are extracted through multiple convolutional blocks;

[0130] S202, input the deep semantic features into the decoding path of the segmentation branch, perform an upsampling operation to restore the feature resolution, and generate upsampled features;

[0131] S203, in the decoding path, the shallow features and the upsampled features are fused by skip connections to form fused features;

[0132] S204, Perform a convolution operation on the fused features to generate a preliminary segmentation result;

[0133] S205, Apply an activation function to the preliminary segmentation result to generate foreground and background segmentation masks.

[0134] In this embodiment, after multi-scale features enter the encoding path of the segmentation branch, they first pass through multiple convolutional blocks to extract shallow and deep semantic features. Shallow features emphasize local edges and fine-grained textures. The convolutional kernels have small receptive areas, and the stride is set to maintain a high-resolution grid. Normalization and nonlinear activation stabilize the amplitude and improve separability. To reduce the impact of noise, lightweight denoising convolutions or learnable channel recalibration can be added after early convolutions to enhance local contrast without disrupting structural coherence. Deep semantic features obtain a larger receptive field and stronger contextual expression by stacking deeper convolutional blocks. Stride or hole rate configurations expand cross-regional dependencies, and residuals or dense structures maintain gradient smoothness and preserve cross-layer information. The encoding path thus simultaneously holds two complementary representations: shallow features preserve high-frequency boundaries, and deep semantic features characterize regional consistency and global layout.

[0135] Deep semantic features are upsampled upon entering the decoding path to restore feature resolution, generating upsampled features. Upsampling can employ transposed convolution, interpolation combined with 1x1 convolution, or subpixel rearrangement; the specific choice is influenced by hardware and latency constraints. To reduce checkerboard artifacts, the stride and padding of the transposed convolution are strictly aligned with the encoding downsampling, or a small kernel convolution is applied after interpolation for spectral compensation. The upsampled features spatially return to the same grid density as the shallow features, laying the foundation for subsequent cross-layer alignment.

[0136] Skip connections are set in the decoding path to align and fuse shallow features with upsampled features, forming fused features. Alignment uses interpolation or lightweight alignment convolutions to ensure spatial size compatibility with channel dimensions, followed by aggregation. Aggregation can use a one-to-one convolution after channel concatenation to achieve channel compression and establish cross-channel interaction, or use element-wise summation to establish a simple phase-consistent fusion, or introduce adaptive weighting with learnable weights to adjust the contributions of shallow and upsampled features according to the scene. To avoid edge jitter caused by phase deviation between scales, small kernel correction convolutions are added to both feature paths before fusion, along with normalization, to ensure that the amplitude range and statistical distribution are within the same dimension. The fused features combine boundary sharpness and regional semantic consistency, adapting to subsequent discrimination calculations.

[0137] Convolutional operations are applied to the fused features to generate preliminary segmentation results. This convolution typically uses a 1x1 or little-kernel configuration, mapping high-dimensional channels to two-channel or single-channel foreground and background probability maps, depending on the task. To improve detail recovery, one or two little-kernel convolutions can be stacked before this convolution, and a dilation rate can be introduced to balance local and contextual aspects. Saturation is then suppressed through normalization and nonlinearity. The preliminary segmentation results express pixel attribution tendencies as continuous probability values, making them suitable for use in conjunction with upstream loss calculations or post-processing.

[0138] The initial segmentation results are converted into foreground and background segmentation masks using an activation function. For binary classification, a sigmoid function is used to map real numbers to the zero-to-one interval, and binarization is performed using a fixed or adaptive threshold. If a two-channel representation is used, Softmax is used to obtain the normalized probability and select the class with the higher probability. The threshold can be optimized through statistical metrics during the validation phase, or it can be adaptively fine-tuned at runtime based on the statistical analysis of the image histogram or connected component area, thus maintaining stable binary output under different exposure or material conditions. The obtained foreground and background segmentation masks correspond one-to-one with the multi-scale features in space, allowing for direct consumption of subsequent spatial weighting and background embedding calculations without additional coordinate transformations. Throughout the entire process, each named entity is generated, transmitted, and used: multi-scale features enter the encoding path to produce shallow and deep semantic features; deep semantic features are upsampled through the decoding path to become upsampled features; skip connections fuse shallow and upsampled features to obtain fused features; convolution operations provide the initial segmentation results; and the activation function outputs the foreground and background segmentation masks. The tensor shape remains strictly aligned with the target mesh at each stage, and convolution and interpolation employ differentiable operations, facilitating end-to-end training and inference sharing.

[0139] This embodiment separates shallow features and deep semantic features through the encoding path and reconstructs spatial resolution through skip connections and upsampling in the decoding path. The fused features have both clear boundaries and robust semantics. The preliminary segmentation results achieve a balance between small object boundaries and large region consistency. The foreground and background segmentation masks after activation function transformation remain stable and usable under different resolutions and imaging conditions.

[0140] In one embodiment, step S30 above includes:

[0141] S301, Extract the background region confidence score from the foreground and background segmentation mask to generate a background confidence matrix;

[0142] S302, Multiply the background confidence matrix element-wise with the multi-scale features to generate a weighted feature map;

[0143] S303, perform background region feature enhancement processing on the weighted feature map, suppress the foreground region feature response, and generate background weighted features.

[0144] In this embodiment, the foreground and background segmentation mask provides pixel-by-pixel background confidence. The background channel probability can be directly obtained, or the background probability can be obtained by subtracting a single channel probability, resulting in a background confidence matrix consistent with the input grid. To improve the usability of the confidence, numerical stabilization and calibration are performed first. A typical process involves adding a minimal constant to the probability to avoid zero gradients, then linearly stretching according to the image quantiles to expand the effective dynamic range, followed by a Gaussian or guided filter for slight smoothing near the boundaries, preserving boundary positions while reducing isolated noise. Considering the need for multi-scale alignment, the background confidence matrix is ​​bilinearly downsampled or upsampled according to the spatial resolution of the feature maps at each scale. The sampling process remains differentiable from the training computation graph, and a one-to-one convolution and normalization are performed after scale switching to correct distribution bias introduced by interpolation. If a binary mask is used, temperature scaling or power-law transformation can be introduced between the binary representation and the probability, transitioning the weights from hard gating to soft gating, facilitating subsequent multiplicative weighted stable convergence.

[0145] When multiplying multi-scale features element-wise with the background confidence matrix, a broadcast mechanism is employed in the channel dimension. This involves expanding the single-channel weights to a tensor consistent with the number of channels, and then performing pixel-wise multiplication at each scale. To balance detail and context, a higher weight steepness is used for high-resolution scales to enhance differences near boundaries, while a gentler weight curve is used for low-resolution scales to preserve consistency across large regions. Multiplicative weighting generates a weighted feature map. Before multiplication, the multi-scale features are aligned in magnitude using normalization based on channel mean and variance or normalization based on channel maximum value to avoid saturation caused by weight and magnitude coupling. After multiplication, the weighted feature map undergoes dynamic range pruning and small bias back-injection to prevent regions with near-zero weights from becoming completely inactive during subsequent propagation. To reduce false suppression, a residual bypass is introduced during multiplication, adding the original features to the weighted result with minimal coefficients. This preserves learnable micro-foreground information while using background confidence as the primary factor for gating.

[0146] Background feature enhancement processing employs two complementary operations on the weighted feature map. The first is background channel enhancement, using a lightweight combination of channel attention and spatial attention. Channel attention generates channel statistics through global average pooling and obtains channel weights using two 1x1 convolutions. Spatial attention generates spatial weights through average and maximum mappings along the channels. The multiplication of these two operations applies to the weighted feature map, promoting enhanced responses in background-related channels and locations. The second is foreground suppression compensation, applying an adaptive suppression strategy to regions with low background confidence. Specifically, a gating factor is introduced into the weighted feature map, obtained by subtracting background confidence from a smoothing threshold function, used to weaken residual high foreground responses. Simultaneously, a small amount of high-frequency enhancement is added to regions with high background confidence and sparse texture, recovering weak structural cues through 3x3 small-kernel convolutions or Laplacian differentiation operations, avoiding over-smoothing. The enhancement stage provides cross-scale consistency constraints, aligning the enhancement results at each scale from low to high in ascending order and weighted summing and back-injecting them to the high-resolution scale, allowing broad-area background consistency to guide the detailed scale. The final output is a background weighted feature with a shape consistent with the highest resolution scale, a stable numerical distribution, and a propagable gradient. Subsequent pooling and normalization can be used directly without additional recalibration.

[0147] This embodiment achieves pixel-level soft gating by constructing a background confidence matrix in the probability space and multiplying it element-wise with multi-scale features. This continuously amplifies background-related information at each scale while weakening foreground-related interference. After weighting, joint channel and spatial enhancement and adaptive suppression are introduced to further improve the signal-to-noise ratio of the background representation and maintain cross-scale consistency. Compared to performing global weighting only once at the end, this process reduces background leakage in the boundary neighborhood, improves consistency in large areas, and restores weak structures in areas with sparse texture. The output background weighted features provide a more concentrated and separable representation for subsequent embedding calculations, thereby reducing the impact of mis-negative samples in contrastive learning and improving retrieval stability.

[0148] In one embodiment, step S40 above includes:

[0149] S401, Perform a global average pooling operation on the background weighted features to generate pooled features;

[0150] S402, Perform a normalization operation on the pooling features to generate a unit vector;

[0151] S403, the unit vector is used as the background embedding vector;

[0152] S404, add the background embedding vector to the first-in-first-out queue of the dynamic memory;

[0153] S405, when the current storage capacity of the dynamic memory reaches the preset capacity, remove the earliest added background embedding vector;

[0154] S406, the background embedding vector stored in the dynamic memory is updated using a moving average strategy.

[0155] In this embodiment, the global average pooling operation performs numerical aggregation along the spatial dimension of the background weighted features, averaging the two-dimensional responses of each channel in both the height and width directions to output a one-dimensional channel vector. The input tensor is denoted as... Pooling result is The calculation formula is To improve stability, outlier values ​​can be pruned before averaging, or a masked mean can be used to exclude low-confidence regions. For background weighted features from multiple scales, scale alignment and summation by scale weights can be performed before a global average pooling operation to maintain statistical consistency. In practice, the global average pooling layer of a deep learning framework can be directly called, ensuring the continuity and differentiability of the backpropagation path.

[0156] After pooling features are formed, a normalization operation is performed to standardize the vector length to a unit radius, resulting in a unit vector. A common implementation is L2 normalization: given p, the norm is first calculated. Next, calculate u = p / ‖p‖2, where ∈ is a minimal constant to avoid division by zero caused by zero vectors. If it is necessary to suppress the dominance of a single channel, channel-wise balancing (e.g., through learnable diagonal scaling) can be performed before L2 normalization. In batch processing scenarios, normalization should be performed independently for each sample to avoid coupling between different samples.

[0157] The unit vector serves as the background embedding vector, representing the global semantic state of the background in a single image. To facilitate downstream similarity matching, a fixed dimension C can be chosen and maintained during training. To reduce storage overhead, orthogonal projection or product quantization encoding can be appended after embedding generation, but it must be ensured that the projection matrix is ​​optimized together with the upstream network during training or fixed and fitted offline. The background embedding vector is stored in memory in floating-point or compressed fixed-point format; if fixed-point quantization is used, the scaling factor should be recorded before writing to restore the numerical range during the decoding stage.

[0158] A dynamic memory stores the time-series background embedding vectors, using a first-in, first-out (FIFO) queue as the data structure. The enqueue process involves packaging the calculated background embedding vector and its metadata (timestamp, source identifier, optional quality score) into an entry and appending it to the tail of the queue. To ensure capacity control, the queue maintains a preset capacity N. When the current storage capacity reaches the limit, a dequeue operation is performed, removing the oldest entry from the front of the queue. The capacity N is related to retrieval throughput and the variance of the similarity distribution. An initial value can be provided through offline statistics, and the capacity can be adaptively adjusted using similarity coverage as an indicator in the training logs. It is recommended to use a circular buffer during implementation, with enqueueing and dequeueing covering at a fixed address to reduce memory allocation overhead.

[0159] A moving average strategy is used to maintain a time-robust representation of entries online. A momentum representation is maintained for each entry in the queue. When a new, same-origin update vector z arrives (e.g., the latest embedding from the same background source), perform... Where m∈[0,1) is the momentum coefficient. If the system cannot reliably identify homology relationships, it can degenerate into performing single-step smoothing only on newly enqueued items, i.e., replacing them with the local mean of the nearest time window. The momentum coefficient can be adaptively adjusted according to time decay or quality score; for high-quality samples, m is reduced to accelerate convergence, and for low-quality samples, m is increased to reduce perturbation. To avoid norm drift, L2 normalization is performed again after momentum update to ensure that all embeddings in the dynamic memory are on the same hypersphere, which facilitates subsequent cosine similarity measurement. In addition, to reduce the instantaneous impact of queue updates on retrieval, a double-buffered write strategy can be adopted: batch momentum updates and normalization are completed in the shadow buffer, and then all at once they are exchanged to the online buffer.

[0160] This embodiment compresses spatial redundancy through global average pooling, aggregates channel statistics after background gating, and outputs a vector proportional to the area and intensity of the background region. L2 normalization restricts the embedding to a unit hypersphere, making the cosine similarity a monotonic function of Euclidean distance, which facilitates stable retrieval. First-in-first-out queues and capacity control ensure that the representation distribution covers the latest scene without losing temporal diversity. Moving average smooths fluctuations from a single source and suppresses occasional noise in the time dimension. The resulting background embedding vector is both discriminative and temporally stable, providing a feature foundation that is scale-consistent, numerically stable, and can be accessed with high throughput for subsequent similarity matching.

[0161] In one embodiment, step S50 above includes:

[0162] S501, determine the cosine similarity value between the background embedding vector and each historical background embedding vector in the dynamic memory;

[0163] S502, obtain the lower similarity threshold and upper similarity threshold for the current training cycle;

[0164] S503, filter out historical background embedding vectors whose cosine similarity value is greater than the lower similarity threshold and less than the upper similarity threshold;

[0165] S504 combines the selected historical background embedding vectors to form a negative sample set;

[0166] S505, The similarity lower limit threshold and the similarity upper limit threshold are updated using an exponential moving average strategy;

[0167] S506, the updated similarity lower threshold and similarity upper threshold are used as the similarity lower threshold and similarity upper threshold for the next training cycle.

[0168] In this embodiment, during training, to select discriminative negative samples from the dynamic memory, it is necessary to calculate the similarity between the background embedding vector and the historical background embedding vectors in the memory, and to select the negative sample set through a dynamically adjusted dual-threshold mechanism. Specifically, the background embedding vector first needs to be normalized to maintain a uniform scale in Euclidean space. This normalization process avoids interference from differences in the magnitude of different feature vectors in the similarity calculation. Assume the current input background embedding vector is z. q The historical context embedding vectors stored in the memory bank are a set Then normalize them respectively to and The normalization formula is:

[0169]

[0170] Based on this, cosine similarity is used as a metric to calculate the similarity between the query vector and each historical vector in the memory. The definition of cosine similarity is:

[0171]

[0172] This metric reflects the similarity of two embedding vectors in direction, avoiding the influence of differences in magnitude. For batch training scenarios, multiple query vectors can be stacked into a matrix. Historical vectors stacked into a matrix The similarity matrix is ​​written as:

[0173]

[0174] After obtaining the similarity distribution, a lower threshold and an upper threshold need to be determined. The lower threshold is used to exclude samples that differ too much from the current background embedding vector, as they provide limited information in contrastive learning. The upper threshold is used to filter out interference terms that are too close to positive samples, as these samples often cause gradient oscillations and instability during training. The range between the two thresholds represents the truly valuable candidate negative samples for learning.

[0175] To avoid significant fluctuations in the threshold during training, an exponential moving average is used to update it. This update method smooths out dynamic changes in the sample distribution during training, allowing the threshold to adapt to new data trends while retaining stable past statistical results. The updated threshold serves as a reference for the next training cycle, ensuring the continuity and stability of the training process.

[0176] In the specific negative sample set construction stage, the system selects all historical background embedding vectors whose similarity falls within the dual threshold range and combines them to form a new negative sample set. If the number of selected samples is too small, the system will automatically expand the threshold range to ensure a sufficient number of negative samples; if the number of selected samples is too large, the system will control the number through balanced sampling or binning strategies to ensure a balance between training efficiency and sample diversity.

[0177] In a distributed training environment, different computing nodes may independently calculate similarity distributions. To avoid inconsistencies in thresholds due to differences between nodes, it is necessary to globally aggregate the statistical results from all nodes before updating the thresholds twice. This ensures that all training nodes use the same selection criteria in a multi-machine, multi-GPU environment, avoiding imbalanced training caused by differences in criteria.

[0178] Ultimately, the negative sample set is stored as an index, and sample extraction is deferred until actually needed. This approach reduces storage and computational overhead and allows for efficient retrieval when required, ensuring smooth computation of the comparative loss during training.

[0179] For example, given the query embedding z q Its negative sample set N(z) q ) Filtering by dual thresholds:

[0180]

[0181] The threshold is updated using an exponential moving average:

[0182]

[0183] Where M is the memory bank, which stores the FIFO queue of historical embeddings; the initial values ​​of τ are set to 0.2 and 0.8 to avoid getting trapped in local optima in early training; T is the transpose of the vector; n is the nth negative sample in the negative sample set; t is the current update step, and t-1 is the update step of the previous time step.

[0184] This embodiment employs a dynamically adjusted dual-threshold mechanism, ensuring that negative sample selection remains adaptive throughout the training cycle. Samples with excessively low similarity are effectively excluded, preventing ineffective learning; samples with excessively high similarity are reasonably removed, preventing the model from getting bogged down in overly difficult comparison tasks. The introduction of an exponential moving average strategy makes threshold changes smoother, avoiding training oscillations caused by excessive batch differences. Through dynamic adjustment and global consistency mechanisms, the selected negative sample set is not only reasonably sized but also diverse in distribution, providing a stable and effective training signal.

[0185] In one embodiment, step S60 above includes:

[0186] S601, Perform a random data augmentation operation on the input image to generate an enhanced image;

[0187] S602, The enhanced image is input into a shared encoder for feature extraction to obtain enhanced multi-scale features;

[0188] S603, Based on the enhanced multi-scale features, an enhanced segmentation mask is generated through segmentation branches;

[0189] S604, Based on the enhanced segmentation mask and enhanced multi-scale features, a positive sample embedding vector is generated through an embedding branch;

[0190] S605, determine the segmentation loss value and cross-entropy loss value based on the foreground and background segmentation mask and the corresponding supervision mask;

[0191] S606, Based on the background embedding vector, the positive sample embedding vector, and the negative sample set, and in combination with the temperature coefficient, determine the contrast loss value;

[0192] S607, determine the mixed loss value based on the segmentation loss value, cross-entropy loss value, and contrast loss value;

[0193] S608, based on the hybrid loss value, update the parameters of the shared encoder, segmentation branch and embedding branch through backpropagation.

[0194] In this embodiment, the process revolves around a closed loop: "data augmentation—positive sample embedding vector—negative sample set—mixed loss—parameter update." The goal is to simultaneously improve the robustness and consistency of the shared encoder, segmentation branch, and embedding branch across both background discrimination and structural segmentation paths. First, random data augmentation is performed to generate enhanced images. The augmentation strategy introduces both geometric perturbations and covers illumination and imaging domain shifts, thereby expanding the distribution without disrupting scene semantics and foreground / background relationships. Geometric perturbations can employ random scaling, random cropping and padding, horizontal flipping, and lightweight rotation; appearance perturbations can use brightness and contrast perturbations, color jitter, fine-grained Gaussian noise, and slight blurring. The augmentation intensity and combination are determined according to hierarchical sampling to avoid excessive perturbation causing semantic drift in the labels. The enhanced image is then input into the shared encoder to extract enhanced multi-scale features, maintaining the same feature hierarchy and channel organization as the non-enhanced path to ensure seamless morphological reuse in subsequent branches. Feature normalization and scale alignment are performed at the encoding end, reducing statistical drift caused by different augmentation strategies.

[0195] Enhanced multi-scale features are fed into the segmentation branch to generate an enhanced segmentation mask. The encoding and decoding paths maintain the structure and connection methods described in the claims above. The upsampling stage strictly aligns with the spatial resolution of the enhanced image, skip connections provide stable boundary cues, and the convolutional prediction layer outputs a probability map or binary map related to the background. To ensure consistency between embedding generation and spatial localization, the enhanced segmentation mask does not need to be coupled with the supervision mask of the training main line; it only plays the role of spatial selection and weight guidance for enhanced samples. The embedding branch receives the enhanced multi-scale features and the enhanced segmentation mask, uses background region weights for spatial convergence and channel compression, and outputs a positive sample embedding vector. This embedding undergoes normalization before entering contrastive learning, making the vectors of different samples comparable in scale, facilitating directional similarity comparison with the negative sample set.

[0196] The semantic segmentation loss term takes the foreground and background segmentation masks and their corresponding supervision masks as inputs to calculate the segmentation loss and cross-entropy loss values ​​used for structure learning. To avoid gradient bias caused by an imbalance in the foreground / background class ratio, the segmentation loss term can use adaptive weights for the background and foreground, or weight key boundaries and weak regions based on pixel confidence, thereby allowing the segmentation branch to obtain more stable convergence signals at structural boundaries, hole regions, and occlusion overlap locations. The cross-entropy term uses the information difference between the supervision mask and the prediction distribution to encourage consistency in pixel-level decision distribution; the combination of the two can balance region overlap and classification probability consistency. To maintain terminology consistency, the inputs to the two loss terms are the segmentation mask and the supervision mask, respectively, and the prediction end uses an output head isomorphic to the enhancement path to avoid representation detours during training.

[0197] The contrastive loss function uses background embedding vectors, positive sample embedding vectors, and a set of negative samples as inputs to perform difficulty-adaptive metric learning. Background embedding vectors are derived from non-enhanced paths and added to a dynamic memory, positive sample embedding vectors are from enhanced paths, and the negative sample set is obtained through a dual-threshold filtering mechanism. When calculating similarity, all embeddings are normalized, and a temperature coefficient is used to adjust the steepness of the similarity distribution, thereby controlling the difficulty of separating positive and negative pairs and the gradient scale. To avoid the impact of "pseudo-negative samples" on stability, soft suppression weights are introduced in the similarity near-threshold region, reducing the gradient contribution of negative samples falling into the critical interval while preserving their shaping effect on the representation boundary. If necessary, background region confidence weights are injected into the contrastive term, allowing high-confidence background regions to dominate the metric signal, further reducing noise caused by foreground leakage.

[0198] The hybrid loss value is obtained by aggregating the segmentation loss value, cross-entropy loss value, and contrastive loss value. During aggregation, learnable or pre-set weight coefficients are used to balance structural consistency and representational discriminativity. The initial values ​​and dynamic adjustment strategies of the weights are related to the dataset size, the degree of class imbalance, and the drift of the memory distribution. To suppress gradient noise of contrastive terms in the early training phase, a warm-up curve is used to gradually increase the contrastive weights in the initial training stage; once the segmentation accuracy reaches a stable range, the contrastive weights are further increased to enhance the discriminative power of the background representation. Backpropagation updates the shared encoder, segmentation branch, and embedding branch jointly. The gradient of the shared encoder comes from both the structural path and the metric path. To avoid gradient cancellation or over-amplification, gradient clipping and normalization are used, and fine-tuning switches for grouping or channel attention are enabled in key residual layers to improve sensitivity to fine-grained background texture. The segmentation branch focuses on the separability of boundaries and large structures, while the embedding branch focuses on the discriminability of background textures, repeating patterns, and material properties. Joint updates establish implicit collaborative constraints in mid-to-high-level features through a shared base, ensuring that both objectives converge towards a "segmentable and searchable" representation space. The optimizer and learning rate scheduling support multi-GPU synchronization, ensuring consistency with the sampling statistics of the dynamic memory. The loss term is numerically aggregated along the data parallel dimension to eliminate small-batch statistical fluctuations.

[0199] It can be implemented in single-card or multi-card environments. In multi-card scenarios, the random seeds for data augmentation are independent within a card but different across cards, and image augmentation and multi-scale feature augmentation are completed locally; the memory index on which the negative sample set depends is aggregated through global communication to ensure that the contrastive items see consistent negative sample statistics. The augmentation strategy can adopt a two-stage pipeline: image-level perturbation first, followed by feature-level perturbation, so that the embedding branch learns a background description with a stable response to local distortion and color shift without increasing inference complexity. The weights of the segmentation loss and cross-entropy loss can be kept relatively high in the first quarter of training to accelerate structural convergence; thereafter, the contrastive loss weight is gradually increased to improve the separability of the background embedding vector. The temperature coefficient can be fine-tuned according to the training process: when the similarity distribution of the negative sample set narrows, the temperature is slightly reduced to amplify subtle differences; when the quality and quantity of negative samples improve, the temperature is increased to stabilize the gradient and prevent gradient bursting. The optimizer can be momentum-based or adaptive, combined with cosine annealing or multi-stage stepped learning rate scheduling. To avoid the embedding branch converging too quickly and limiting the encoder, a smaller learning rate or delayed unfreezing strategy can be introduced into the embedding head early on. Hybrid precision training and gradient accumulation are used to maintain a sufficient batch size under limited GPU memory to stabilize the statistics of the contrast terms. To improve robustness, threshold jitter can be added to the process of "enhancing the selection of background regions by the segmentation mask" to mitigate the propagation of uncertainty of boundary pixels into the embedding space. Mild deduplication and time decay are introduced into the dynamic memory sampling to increase the diversity of negative samples and reduce the dominance of old samples. If there are significant differences in the imaging domain in the application domain, such as different cameras or different lighting environments, small perturbations of style transfer or color mapping can be inserted into the enhancement strategy to explicitly expose the domain shift during training and reduce the performance drop-off during deployment.

[0200] For example, samples are divided into positive and negative pairs. Positive pairs are obtained through data augmentation (rotation, flipping, cropping, Gaussian blurring), while negative pairs are obtained from a memory bank through dual-threshold filtering. The initial memory bank calculates the inner product between data feature vectors (obtained by the ResNet-50 backbone network and the segmentation branch F⊙S) using an enumeration method on the original dataset. The inner product values ​​are then placed into a FIFO queue in the memory bank from smallest to largest, with the queue size set. During model training, each data feature is updated, resulting in new negative sample data updating the database. With the positive and negative pairs in place, contrastive learning training is performed, using a hybrid loss function instead of the traditional InfoNCE loss function to train the model parameters.

[0201] The total loss consists of three parts:

[0202] L=λ1L dice +λ2L ce +λ3L cont

[0203] This indicates that a weighted combined loss function is used during training, which consists of three parts: Dice loss, cross-entropy loss, and contrastive loss. λ1, λ2, and λ3 are the weight coefficients of the three losses, used to adjust the contribution of the three types of losses to the overall optimization. Experiments show that the best results are achieved when λ1 = λ2 = 1 and λ3 = 0.5.

[0204] The segmentation loss is a weighted sum of the Dice coefficient and the cross-entropy:

[0205]

[0206] Dice loss is primarily used to measure the degree of overlap between the predicted mask and the true mask.

[0207] S gt represents the ground truth mask, with a value of 0 or 1, indicating whether a pixel belongs to the target (background or foreground). S represents the probability mask predicted by the model, with a value between [0,1]. 2∑S gt S represents the number of pixels that intersect with the predicted values ​​(weighted similarity). ∑S gt +∑S represents the sum of predicted pixels and actual pixels. The closer the Dice value is to 1, the better the prediction result matches the actual segmentation result.

[0208] Cross-entropy loss L ce :

[0209] L ce =-∑[S gt logS+(1-S gt log(1-S)]

[0210] Cross-entropy is used to measure the difference between the predicted probability distribution and the true label:

[0211] S gt is the actual label mask (manually labeled). S is the predicted mask probability. gt logS represents the prediction accuracy for the positive class (target region). [S] gt logS+(1-S gt [log(1-S)] represents the prediction accuracy for the negative class (non-target region).

[0212] Comparison loss L cont :

[0213]

[0214] Among them, z q This represents the query embedding. pThis represents the positive embedding vector, generated from the same image or augmented samples. n ∈N represents the set of negative sample embedding vectors. σ represents the temperature parameter, which controls the smoothness of the similarity distribution. A temperature coefficient σ = 0.07 is introduced for the contrast loss. This indicates the similarity between the query and the positive sample. It includes the sum of similarities between positive samples and all negative samples.

[0215] By maximizing the proportion of positive sample similarity in the overall distribution, the discriminative learning of positive and negative samples can be achieved.

[0216] Example Description: In the healthcare field, complex medical image background segmentation and retrieval tasks can be achieved, thereby improving diagnostic efficiency and the reliability of decision support. First, in the input stage, the system acquires medical images, such as CT slices, MRI sequences, or ultrasound images, and performs size standardization on the input images to eliminate differences caused by varying device resolutions and sampling intervals. The standardized medical images are then fed into a shared encoder, where basic texture features are extracted through an initial convolutional layer. Following this, multiple feature extraction stages gradually generate feature maps at different scales. Finally, the multi-scale features are fused to simultaneously preserve vascular details, organ boundaries, and overall anatomical structure.

[0217] Next, the fused multi-scale features are input into the segmentation branch. In the encoding path of this branch, convolutional blocks extract shallow and deep features layer by layer. Shallow features contain rich local details such as tumor edges, while deep semantic features carry the global relationship between tissue category and lesion background. The decoding path restores resolution through progressive upsampling and combines shallow detail information with upsampled features using skip connections to obtain more complete segmentation information. Based on this, convolutional layers further refine the boundaries, and finally, an activation function is used to obtain foreground and background segmentation masks, where the foreground represents the target lesion or abnormal region, and the background corresponds to healthy tissue or non-target regions.

[0218] The generated foreground and background segmentation masks are used to perform spatial weighting on the multi-scale features. In medical images, lesions often occupy only local areas; therefore, the confidence of the background region is extracted through the mask to form a background confidence matrix. This matrix is ​​multiplied element-wise with the original multi-scale features to obtain a weighted feature map that suppresses foreground interference. Subsequently, background region enhancement is performed on the weighted features to ensure that non-lesion features around blood vessels, soft tissues, or organs are more prominently preserved, thereby generating background-weighted features.

[0219] After obtaining the background-weighted features, the system performs global average pooling to compress the features of different spatial dimensions into fixed-length representations, and then normalizes them into unit vectors to form the background embedding vector. This embedding vector is added to a dynamic memory, which uses a first-in-first-out queue structure to ensure that the most recent background information is retained first. When the memory reaches its capacity limit, the oldest vector is removed, and the stored vectors are updated using a moving average strategy, so that the entire memory always maintains a dynamic medical background representation.

[0220] During model optimization, a negative sample set needs to be selected to construct the contrastive learning target. By calculating the cosine similarity between the current background embedding vector and the historical background embedding vectors in the memory bank, and combining this with a dynamically adjusted dual-threshold mechanism, difficult negative samples that are too close to positive samples and easy negative samples that are too simple are excluded. Embedding vectors within the threshold range are selected to form the negative sample set. This mechanism ensures that the negative samples in the learning process are both discriminative and do not lead to training instability.

[0221] To further enhance the model's generalization ability, augmentation operations such as rotation, flipping, and random cropping are applied to the input medical images to obtain augmented images, which are then input into the shared encoder and segmentation branch to generate augmented segmentation masks and multi-scale features. Based on these augmented features, positive sample embedding vectors are obtained through the embedding branch. Then, the segmentation loss and cross-entropy loss are calculated using the original segmentation mask and a manually labeled supervised mask. Simultaneously, the contrast loss is calculated by combining the background embedding vector, positive sample embedding vector, and negative sample set with a temperature coefficient. These three are weighted and summed to form a hybrid loss value. Backpropagation is used to optimize the parameters of the shared encoder, segmentation branch, and embedding branch, enabling the model to gradually learn a more robust background representation.

[0222] After iterative training, the updated shared encoder, segmentation branch, and embedding branch are applied to new query medical images. The system extracts multi-scale features from the query image, generates a query segmentation mask, obtains query background weighted features through spatial weighting, and generates a query background embedding vector through pooling and normalization. This vector is matched with background embedding vectors in the dynamic memory for similarity, and sorted by cosine similarity to obtain the top few similar background images, forming a search results list. Doctors can quickly retrieve historical samples similar to the target case in background tissue features for auxiliary diagnosis or lesion identification.

[0223] The value of this process in healthcare settings lies in the fact that doctors often need to compare numerous case backgrounds when faced with complex images. Automated background feature retrieval can provide the most similar reference image within seconds, reducing the burden of manual comparison and revealing potential structural similarities, thus improving diagnostic consistency and efficiency. For example, in liver CT images, the system can automatically retrieve cases with similar vascular distribution and tissue texture, providing background comparison for the diagnosis of liver cancer or liver fibrosis. In lung CT screening, it can also quickly match similar lung background patterns, thereby helping to distinguish lesions from normal variations and reducing the misdiagnosis rate.

[0224] In the fintech business, the system enables background modeling and retrieval of complex transaction data and scene images, thereby improving the accuracy and real-time performance of risk control systems, anti-fraud detection, and transaction behavior analysis. First, in the data input stage, the system receives input images from the transaction scene, such as ATM monitoring screens, screenshots of mobile payment interfaces, or video frames from financial outlets. Due to differences in resolution and sampling conditions across different terminals, the system first performs size standardization processing, unifying the original images to a preset input size. The standardized images are then fed into a shared encoder, where basic texture features, such as interface elements, document borders, and background textures of the transaction environment, are extracted through initial convolutional layers. Subsequently, through multiple feature extraction stages, feature maps at different scales are generated, capturing both the details of document text and preserving the macroscopic structure of the scene environment. Finally, multi-scale features are fused to form a comprehensive representation.

[0225] Next, the fused features are fed into the segmentation branch. In the encoding path, convolutional blocks extract shallow and deep information. Shallow features are used to preserve the local structure of the interface and the edges of the ticket, while deep semantic features cover the overall category of the transaction scene and the background context. The decoding path gradually restores the spatial resolution through upsampling, and simultaneously uses skip connections to fuse the shallow and upsampled features, obtaining fused features that contain details and semantics. Subsequently, convolution is used to further refine the boundaries, and an activation function is combined to output a segmentation mask for the foreground and background. The foreground can be the user, ticket, or transaction interface elements, while the background is the environmental scene or irrelevant information.

[0226] Multi-scale features are spatially weighted using segmentation masks to extract the background region confidence matrix. This matrix is ​​then multiplied element-by-element with the features to obtain a weighted feature map. This process weakens background interference around the ticket (such as desktop texture or lighting variations) while preserving true background patterns (such as counter environment and terminal shell features). Subsequently, background region feature enhancement is performed to strengthen background features related to the transaction environment, generating weighted background features.

[0227] Next, the background weighted features are subjected to global average pooling, compressed into a fixed-length representation, and then normalized to form a unit vector, resulting in the background embedding vector. This embedding vector is stored in a dynamic memory, which uses a first-in, first-out (FIFO) mechanism to maintain data freshness and is continuously updated through a moving average to ensure that the memory reflects the latest transaction background patterns. For example, when an ATM is redesigned or the payment interface is upgraded, the system can update the memory in a short time to adapt to the new background features.

[0228] To ensure the stability of contrastive learning, the system calculates the distance between the current background embedding vector and historical background embedding vectors using cosine similarity. Combined with a dynamic dual-threshold mechanism, it filters the negative sample set, removing extremely difficult negative samples that are too close to positive samples and extremely easy negative samples that lack information, retaining background embedding vectors of moderate difficulty as the negative sample set. This filtering method avoids overfitting during training or non-convergence during optimization, enabling the model to stably learn discriminative background features.

[0229] During the training phase, the system applies random data augmentation to the input image, such as rotation, cropping, brightness perturbation, or random occlusion, generating an enhanced image which is then input into the shared encoder and segmentation branch. This yields an enhanced segmentation mask and multi-scale features, and positive sample embedding vectors are generated through the embedding branch. Simultaneously, segmentation loss and cross-entropy loss are calculated using the original segmentation mask and the labeled mask. These are then combined with the background embedding vector, positive sample embedding vector, and negative sample set to calculate the contrastive loss. These three types of losses are synthesized into a hybrid loss, which is used to update the encoder and branch parameters through backpropagation, making the model more robust to feature representations in transaction scenarios.

[0230] During the inference phase, the trained and updated shared encoder, segmentation branch, and embedding branch are used to process new query images, such as a screenshot of a bank card transaction monitoring. The system extracts multi-scale features from the query image, generates a query segmentation mask, obtains query background weighted features through spatial weighting, and then obtains a query background embedding vector through pooling and normalization. This vector is compared with the background embedding vectors in the dynamic memory for similarity calculation and ranking, ultimately forming a list of retrieval results. The system can quickly retrieve transaction scenarios highly similar to the current query background from the historical sample database, such as the same counter environment, the same payment terminal, and similar interface layout, thereby helping to determine whether the transaction scenario is normal or whether there is a fake background.

[0231] This mechanism brings direct value to fintech scenarios. Through dynamically updated background retrieval, risk control systems can quickly detect abnormal environments, such as forged transaction interfaces, tampered ATM interfaces, or fake backgrounds introduced in remote attacks. Combined with background similarity matching, the system can identify the differences between forged and normal transactions, providing reliable feature support for anti-fraud models. For example, in bank card fraud detection, if the background embedding vector differs significantly from the real transaction environment in the dynamic memory, the system can trigger an alert. In remote account opening scenarios, searching the historical background database can also confirm whether the image submitted by the applicant is in a real branch environment, thereby reducing the risk of forged documents or fraudulent account opening.

[0232] This embodiment constructs stable positive sample embedding vectors through a joint update process using data augmentation, provides a set of negative samples of moderate difficulty using a dynamic memory, reinforces structural consistency with segmentation loss and cross-entropy loss, and constrains the discriminativeness of background representation with contrastive loss. The hybrid loss value, which integrates these three factors, establishes a collaborative gradient between the shared encoder, segmentation branch, and embedding branch, promoting the unification of "segmentability" and "retrievalability" in the same representation space. On the one hand, the segmentation path obtains more stable optimized signals in boundary and detail regions, reducing the interference of foreground leakage on background representation; on the other hand, the embedding path forms clearer inter-class intervals on complex textures and similar materials, enabling background retrieval to maintain high consistency and transferability across scenes and devices. At the same time, the training process is more stable due to the phased scheduling of temperature and weights and the threshold smoothing strategy, making it less prone to gradient oscillations and premature convergence.

[0233] In one embodiment, a retrieval device based on foreground and background segmentation masks is provided, which corresponds one-to-one with the retrieval method based on foreground and background segmentation masks described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the retrieval device based on foreground and background segmentation masks of the present invention. The modules include a shared encoder module 10, a segmentation branch module 20, a spatial weighting module 30, an embedding branch module 40, a dynamic memory module 50, a negative sample screening module 60, a query processing module 70, and a similarity retrieval module 80. Detailed descriptions of each functional module are as follows:

[0234] The shared encoder module 10 is used to acquire an input image and input the input image into the shared encoder to obtain multi-scale features;

[0235] The segmentation branch module 20 is used to input the multi-scale features into the segmentation branch to generate foreground and background segmentation masks;

[0236] Spatial weighting module 30 is used to perform spatial weighting processing on the multi-scale features based on the foreground and background segmentation mask to obtain background weighted features;

[0237] The embedding branch module 40 is used to perform pooling and normalization operations on the background weighted features to generate a background embedding vector and add the background embedding vector to the dynamic memory.

[0238] The dynamic memory module 50 is used to filter the negative sample set based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory through dynamically adjusted dual thresholds.

[0239] The negative sample screening module 60 is used to update the shared encoder, segmentation branch and embedding branch based on the positive sample embedding vector generated by data augmentation and the negative sample set.

[0240] The query processing module 70 is used to process the query image using the updated shared encoder, segmentation branch and embedding branch, and generate a query background embedding vector.

[0241] The similarity retrieval module 80 is used to perform similarity matching between the query background embedding vector and the background embedding vector in the dynamic memory to obtain a list of retrieval results.

[0242] In one embodiment, the shared encoder module 10 is specifically used for:

[0243] Acquire an input image, perform size standardization processing on the input image, and generate a standardized input image;

[0244] The standardized input image is input into the initial convolutional layer of the shared encoder to extract primary texture features;

[0245] The primary texture features are input into the first feature extraction stage of the shared encoder to generate a first-scale feature map.

[0246] The first-scale feature map is input into the second feature extraction stage of the shared encoder to generate a second-scale feature map.

[0247] The second-scale feature map is input into the third feature extraction stage of the shared encoder to generate the third-scale feature map;

[0248] The primary texture features, the first-scale feature map, the second-scale feature map, and the third-scale feature map are fused to form a multi-scale feature.

[0249] In one embodiment, the branch splitting module 20 is specifically used for:

[0250] The multi-scale features are input into the encoding path of the segmentation branch, and shallow features and deep semantic features are extracted through multiple convolutional blocks;

[0251] The deep semantic features are input into the decoding path of the segmentation branch, and an upsampling operation is performed to restore the feature resolution and generate upsampled features.

[0252] In the decoding path, the shallow features and the upsampled features are fused by skip connections to form fused features;

[0253] Perform a convolution operation on the fused features to generate preliminary segmentation results;

[0254] An activation function is applied to the preliminary segmentation results to generate foreground and background segmentation masks.

[0255] In one embodiment, the spatial weighting module 30 is specifically used for:

[0256] Extract the background region confidence score from the foreground and background segmentation mask to generate a background confidence matrix;

[0257] The background confidence matrix is ​​multiplied element-wise with the multi-scale features to generate a weighted feature map;

[0258] The weighted feature map is subjected to background region feature enhancement processing to suppress the foreground region feature response, thereby generating background weighted features.

[0259] In one embodiment, the embedded branch module 40 is specifically used for:

[0260] Perform a global average pooling operation on the background weighted features to generate pooled features;

[0261] Perform a normalization operation on the pooled features to generate unit vectors;

[0262] The unit vector is used as the background embedding vector;

[0263] The background embedding vector is added to the first-in-first-out queue of the dynamic memory;

[0264] When the current storage capacity of the dynamic memory reaches the preset capacity, the earliest added background embedding vector is removed;

[0265] The background embedding vector stored in the dynamic memory is updated using a moving average strategy.

[0266] In one embodiment, the dynamic memory module 50 is specifically used for:

[0267] Determine the cosine similarity value between the background embedding vector and each historical background embedding vector in the dynamic memory;

[0268] Obtain the lower similarity threshold and upper similarity threshold for the current training cycle;

[0269] Historical background embedding vectors with cosine similarity values ​​greater than the lower similarity threshold and less than the upper similarity threshold are selected.

[0270] The selected historical background embedding vectors are combined to form a negative sample set;

[0271] The lower and upper similarity thresholds are updated using an exponential moving average strategy.

[0272] The updated lower and upper similarity thresholds will be used as the lower and upper similarity thresholds for the next training cycle.

[0273] In one embodiment, the negative sample screening module 60 is specifically used for:

[0274] Perform random data augmentation on the input image to generate an augmented image;

[0275] The enhanced image is input into a shared encoder for feature extraction to obtain enhanced multi-scale features;

[0276] Based on the enhanced multi-scale features, an enhanced segmentation mask is generated through segmentation branches;

[0277] Based on the enhanced segmentation mask and enhanced multi-scale features, positive sample embedding vectors are generated through embedding branches;

[0278] The segmentation loss value and cross-entropy loss value are determined based on the foreground and background segmentation masks and the corresponding supervision masks;

[0279] The contrast loss value is determined based on the background embedding vector, the positive sample embedding vector, and the negative sample set, combined with the temperature coefficient.

[0280] The mixed loss value is determined based on the segmentation loss value, cross-entropy loss value, and contrastive loss value.

[0281] Based on the hybrid loss value, the parameters of the shared encoder, segmentation branch, and embedding branch are updated through backpropagation.

[0282] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side retrieval method based on foreground and background segmentation masks.

[0283] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a user-side function or step of a retrieval method based on foreground and background segmentation masks.

[0284] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0285] Acquire an input image and input the input image into a shared encoder to obtain multi-scale features;

[0286] The multi-scale features are input into the segmentation branch to generate foreground and background segmentation masks;

[0287] Based on the foreground and background segmentation mask, the multi-scale features are spatially weighted to obtain background weighted features;

[0288] Pooling and normalization operations are performed on the background weighted features to generate a background embedding vector, and the background embedding vector is added to the dynamic memory.

[0289] Based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory, the negative sample set is filtered by dynamically adjusted dual thresholds.

[0290] Based on the positive sample embedding vector generated by data augmentation and the negative sample set, update the shared encoder, segmentation branch and embedding branch;

[0291] The query image is processed using the updated shared encoder, segmentation branch, and embedding branch to generate a query background embedding vector.

[0292] The query background embedding vector is matched with the background embedding vector in the dynamic memory to obtain a list of search results.

[0293] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0294] Acquire an input image and input the input image into a shared encoder to obtain multi-scale features;

[0295] The multi-scale features are input into the segmentation branch to generate foreground and background segmentation masks;

[0296] Based on the foreground and background segmentation mask, the multi-scale features are spatially weighted to obtain background weighted features;

[0297] Pooling and normalization operations are performed on the background weighted features to generate a background embedding vector, and the background embedding vector is added to the dynamic memory.

[0298] Based on the similarity between the background embedding vector and the historical background embedding vector in the dynamic memory, the negative sample set is filtered by dynamically adjusted dual thresholds.

[0299] Based on the positive sample embedding vector generated by data augmentation and the negative sample set, update the shared encoder, segmentation branch and embedding branch;

[0300] The query image is processed using the updated shared encoder, segmentation branch, and embedding branch to generate a query background embedding vector.

[0301] The query background embedding vector is matched with the background embedding vector in the dynamic memory to obtain a list of search results.

[0302] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0303] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0304] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0305] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A foreground versus background segmentation mask based retrieval method, characterized in that, The method comprises the following steps: obtaining an input image, inputting the input image into a shared encoder to obtain multi-scale features; inputting the multi-scale features into a segmentation branch to generate foreground and background segmentation masks; performing spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features; performing pooling and normalization operations on the background weighted features to generate a background embedding vector, and adding the background embedding vector to a dynamic memory bank; based on the similarity between the background embedding vector and the historical background embedding vectors in the dynamic memory bank, filtering a negative sample set through a dynamically adjusted double threshold value; based on the positive sample embedding vectors generated by data augmentation and the negative sample set, updating the shared encoder, the segmentation branch and the embedding branch; processing a query image using the updated shared encoder, segmentation branch and embedding branch to generate a query background embedding vector; performing similarity matching on the query background embedding vector and the background embedding vectors in the dynamic memory bank to obtain a retrieval result list.

2. The foreground and background segmentation mask based retrieval method of claim 1, wherein, obtaining an input image, inputting the input image into a shared encoder to obtain multi-scale features, comprising: obtaining an input image, performing size standardization processing on the input image to generate a standardized input image; inputting the standardized input image into the initial convolution layer of the shared encoder to extract primary texture features; inputting the primary texture features into the first feature extraction stage of the shared encoder to generate a first scale feature map; inputting the first scale feature map into the second feature extraction stage of the shared encoder to generate a second scale feature map; inputting the second scale feature map into the third feature extraction stage of the shared encoder to generate a third scale feature map; fusing the primary texture features, the first scale feature map, the second scale feature map and the third scale feature map to form multi-scale features.

3. The foreground and background segmentation mask based retrieval method of claim 1, wherein, inputting the multi-scale features into a segmentation branch to generate foreground and background segmentation masks, comprising: inputting the multi-scale features into the encoding path of the segmentation branch to extract shallow features and deep semantic features through multiple convolution blocks; inputting the deep semantic features into the decoding path of the segmentation branch to perform upsampling operation to restore feature resolution and generate upsampled features; in the decoding path, the shallow features and the upsampled features are fused through a jump connection to form fused features; performing convolution operation on the fused features to generate a preliminary segmentation result; applying an activation function to the preliminary segmentation result to generate foreground and background segmentation masks.

4. The foreground and background segmentation mask based retrieval method of claim 1, wherein, performing spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features, comprising: extracting background region confidence in the foreground and background segmentation masks to generate a background confidence matrix; performing element-wise multiplication between the background confidence matrix and the multi-scale features to generate a weighted feature map; performing background region feature enhancement processing on the weighted feature map to suppress foreground region feature response to generate background weighted features.

5. The foreground and background segmentation mask based retrieval method of claim 1, wherein, performing pooling and normalization operations on the background weighted features to generate a background embedding vector, and adding the background embedding vector to a dynamic memory bank, comprising: performing a global average pooling operation on the background weighted feature to generate a pooled feature; performing a normalization operation on the pooled feature to generate a unit vector; taking the unit vector as a background embedding vector; adding the background embedding vector to a first-in-first-out queue of a dynamic memory bank; when a current storage capacity of the dynamic memory bank reaches a preset capacity, removing the earliest added background embedding vector; updating the stored background embedding vectors in the dynamic memory bank using a sliding average strategy.

6. The foreground and background segmentation mask based retrieval method of claim 1, wherein, based on the similarity between the background embedding vector and the historical background embedding vectors in the dynamic memory bank, filtering a negative sample set through a dynamically adjusted double threshold value, including: determining the cosine similarity value between the background embedding vector and each historical background embedding vector in the dynamic memory bank; obtaining a similarity lower threshold value and a similarity upper threshold value of the current training period; filtering out the historical background embedding vectors with a cosine similarity value greater than the similarity lower threshold value and less than the similarity upper threshold value; combining the filtered historical background embedding vectors to form a negative sample set; updating the similarity lower threshold value and the similarity upper threshold value using an exponential moving average strategy; taking the updated similarity lower threshold value and the similarity upper threshold value as the similarity lower threshold value and the similarity upper threshold value of the next training period.

7. The foreground and background segmentation mask based retrieval method of claim 1, wherein, based on the positive sample embedding vector generated by data augmentation and the negative sample set, updating the shared encoder, the segmentation branch and the embedding branch, including: performing a random data augmentation operation on the input image to generate an augmented image; inputting the augmented image into the shared encoder for feature extraction to obtain augmented multi-scale features; generating an augmented segmentation mask based on the augmented multi-scale features through the segmentation branch; generating a positive sample embedding vector based on the augmented segmentation mask and the augmented multi-scale features through the embedding branch; determining a segmentation loss value and a cross-entropy loss value based on the foreground and background segmentation masks and the corresponding supervision masks; determining a contrast loss value based on the background embedding vector, the positive sample embedding vector and the negative sample set, and combining a temperature coefficient; determining a hybrid loss value according to the segmentation loss value, the cross-entropy loss value and the contrast loss value; updating the parameters of the shared encoder, the segmentation branch and the embedding branch through back propagation based on the hybrid loss value.

8. A foreground versus background segmentation mask based retrieval apparatus, characterized in that, The retrieval device based on the foreground and background segmentation masks includes: a shared encoder module configured to obtain an input image, input the input image into a shared encoder, and obtain multi-scale features; a segmentation branch module configured to input the multi-scale features into a segmentation branch to generate foreground and background segmentation masks; a spatial weighting module configured to perform spatial weighting processing on the multi-scale features based on the foreground and background segmentation masks to obtain background weighted features; an embedding branch module configured to perform pooling and normalization operations on the background weighted features to generate a background embedding vector, and add the background embedding vector to a dynamic memory bank; a dynamic memory bank module configured to filter a negative sample set through a dynamically adjusted double threshold value based on the similarity between the background embedding vector and the historical background embedding vectors in the dynamic memory bank. The negative sample screening module is configured to update the shared encoder, the segmentation branch, and the embedding branch based on the positive sample embedding vector generated by data augmentation and the negative sample set; The query processing module is configured to process a query image using the updated shared encoder, the segmentation branch, and the embedding branch to generate a query background embedding vector. The similarity retrieval module is configured to perform similarity matching between the query background embedding vector and the background embedding vectors in the dynamic memory bank to obtain a retrieval result list.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a foreground and background segmentation mask based retrieval program stored on the memory and executable on the processor. When the foreground and background segmentation mask based retrieval program is executed by the processor, the steps of the foreground and background segmentation mask based retrieval method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The storage medium stores a foreground and background segmentation mask based retrieval program. When the foreground and background segmentation mask based retrieval program is executed by the processor, the steps of the foreground and background segmentation mask based retrieval method according to any one of claims 1-7 are implemented.

Images