Image contrast model training method, image contrast method, device, and medium

Abstract

This application provides an image comparison model training method, image comparison method, device, and medium, belonging to the field of artificial intelligence technology. The method involves inputting the color, depth, and semantic features of a first image into a feature fusion sub-model to output a first fused feature; inputting the color, depth, and semantic features of a second image into the feature fusion sub-model to output a second fused feature; inputting the first fused feature, first depth feature, and first semantic feature into a background perception sub-model to output a first image feature; inputting the second fused feature, second depth feature, and second semantic feature into the background perception sub-model to output a second image feature; inputting the first image feature and second image feature into a background feature extraction sub-model to output a first background feature and a second background feature, and determining their predicted similarity; iteratively training the model based on the difference between the predicted similarity and the label similarity; the trained model can be applied to the financial and medical fields.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image comparison model training method, image comparison method, device and medium. Background Technology

[0002] In some image processing scenarios, it is necessary to calculate the background similarity between images in order to determine whether two images were taken under the same background.

[0003] In related technologies, the approach is to directly cut out the foreground and background, and then compare the similarity between the backgrounds of different images. However, this approach is relatively simple and cannot be used to process complex images, such as images with dynamic foreground interference (such as people) or images with appearance differences caused by changes in lighting and viewing angles. Related technologies have difficulty in accurately obtaining the background in the image, making it difficult to accurately measure the structural and content similarity of the background areas of two images. Summary of the Invention

[0004] The main objective of this application is to propose an image comparison model training method, image comparison method, device and medium, which can accurately determine the background similarity of two images after training.

[0005] To achieve the above objectives, a first aspect of this application proposes an image contrast model training method, the method comprising: Obtain the first image and the second image, as well as the label similarity between the first image and the second image; Obtain the first color feature, the first depth feature, and the first semantic feature of the first image; obtain the second color feature, the second depth feature, and the second semantic feature of the second image. The first color feature, the first depth feature, and the first semantic feature are input into the feature fusion sub-model to output the first fused feature, and the second color feature, the second depth feature, and the second semantic feature are input into the feature fusion sub-model to output the second fused feature; The first fusion feature, the first depth feature, and the first semantic feature are input into the background perception sub-model to output the first image feature, and the second fusion feature, the second depth feature, and the second semantic feature are input into the background perception sub-model to output the second image feature; The first image feature is input into the background feature extraction sub-model, and the first background feature is output. The second image feature is input into the background feature extraction sub-model, and the second background feature is output. The predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature. The image comparison model is then iteratively trained based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

[0006] To achieve the above objectives, a second aspect of this application proposes an image comparison method, applied to an image comparison model trained based on the image comparison model training method provided in this application, the method comprising: Acquire a first target image and a second target image, as well as a first target color feature, a first target depth feature and a first target semantic feature of the first target image, and a second target color feature, a second target depth feature and a second target semantic feature of the second target image; The first target color feature, the first target depth feature, and the first target semantic feature are input into the trained feature fusion sub-model to output the first target fusion feature. The second target color feature, the second target depth feature, and the second target semantic feature are input into the trained feature fusion sub-model to output the second target fusion feature. The first target fusion feature, the first target depth feature, and the first target semantic feature are input into the trained background perception sub-model to output the first target image feature. The second target fusion feature, the second target depth feature, and the second target semantic feature are input into the trained background perception sub-model to output the second target image feature. The first target image features are input into the trained background feature extraction sub-model to output the first target background features, and the second target image features are input into the trained background feature extraction sub-model to output the second target background features. The target similarity between the first target image and the second target image is determined based on the first target background features and the second target background features.

[0007] To achieve the above objectives, a third aspect of this application proposes an image contrast model training apparatus, wherein the image contrast model includes a feature fusion sub-model, a background perception sub-model, and a background extraction sub-model, comprising: An image acquisition unit is used to acquire a first image and a second image, as well as the label similarity between the first image and the second image; The feature acquisition unit is used to acquire the first color feature, the first depth feature, and the first semantic feature of the first image, and to acquire the second color feature, the second depth feature, and the second semantic feature of the second image; The feature fusion unit is used to input the first color feature, the first depth feature, and the first semantic feature into the feature fusion sub-model and output the first fused feature, and to input the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model and output the second fused feature; The background perception unit is used to input the first fusion feature, the first depth feature, and the first semantic feature into the background perception sub-model and output the first image feature, and to input the second fusion feature, the second depth feature, and the second semantic feature into the background perception sub-model and output the second image feature; The background extraction unit is used to input the first image features into the background feature extraction sub-model and output the first background features, and to input the second image features into the background feature extraction sub-model and output the second background features; The background comparison unit is used to determine the predicted similarity between the first image and the second image based on the first background feature and the second background feature, and to iteratively train the image comparison model based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

[0008] In some implementations, the feature acquisition unit is used for: The first image is input into a pre-trained color feature extraction model, which outputs the first color feature. The first image is input into a pre-trained deep feature extraction model, which outputs the first deep feature. The first image is input into a pre-trained semantic feature extraction model, which outputs the first semantic feature.

[0009] In some implementations, the feature fusion sub-model includes a cross-attention module and a residual module; the feature fusion unit is used for: The first color feature, the first depth feature, and the first semantic feature are dimensionally unified to obtain the first updated color feature, the first updated depth feature, and the first updated semantic feature. The first updated color feature, the first updated depth feature, and the first updated semantic feature are input into the cross-attention module, and the fused attention feature is output. The fused attention features are input into the residual module, and the first fused feature is output.

[0010] In some implementations, the background-aware sub-model includes a first attention module, a second attention module, a gating weight module, and a fusion module; the background-aware unit is used for: The first depth feature is input into the first attention module, and the spatial attention weight corresponding to each pixel in the first image is output. The first semantic feature is input into the second attention module, and the channel attention weights corresponding to the first image are output. The first fusion feature, the spatial attention weight, and the channel attention weight are input into the gating weight module, and the first gating fusion weight is output. The first gating fusion weight and the first fusion feature are input into the fusion module, and the first image feature is output.

[0011] In some implementations, the background contrast unit is used for: Determine the first cosine distance matrix between different feature points in the first background feature, and determine the second cosine distance matrix between different feature points in the second background feature; Determine the similarity of the probability distributions between the first cosine distance matrix and the second cosine distance matrix; Based on the first gated fusion weight and the second gated fusion weight corresponding to the second image, the background space consistency between the first image and the second image is determined; Based on the probability distribution similarity and the background space consistency, the predicted similarity between the first image and the second image is determined.

[0012] In some implementations, the background feature extraction sub-model includes multiple sequentially connected background attention modules; the background extraction unit is used for: The first image features are input into the first background attention module, and the background attention features are output. The background attention features are input into the next background attention module after the first background attention module, and the updated background attention features are output. The background attention features output by the previous background attention module are input into the next background attention module until the last background attention module outputs the first background feature.

[0013] To achieve the above objectives, a fourth aspect of this application provides an image comparison apparatus, applied to an image comparison model trained based on the image comparison model training method provided in this application. The apparatus includes: The target image acquisition unit is used to acquire a first target image and a second target image, as well as a first target color feature, a first target depth feature and a first target semantic feature of the first target image, and a second target color feature, a second target depth feature and a second target semantic feature of the second target image; The target feature fusion unit is used to input the first target color feature, the first target depth feature, and the first target semantic feature into the trained feature fusion sub-model and output the first target fusion feature, and input the second target color feature, the second target depth feature, and the second target semantic feature into the trained feature fusion sub-model and output the second target fusion feature; The target background perception unit is used to input the first target fusion feature, the first target depth feature and the first target semantic feature into the trained background perception sub-model and output the first target image feature, and input the second target fusion feature, the second target depth feature and the second target semantic feature into the trained background perception sub-model and output the second target image feature; The target background extraction unit is used to input the first target image features into the trained background feature extraction sub-model and output the first target background features, and input the second target image features into the trained background feature extraction sub-model and output the second target background features; The target background comparison unit is used to determine the target similarity between the first target image and the second target image based on the first target background feature and the second target background feature.

[0014] To achieve the above objectives, a fifth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first or second aspect above.

[0015] To achieve the above objectives, a sixth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first or second aspect.

[0016] The image comparison model training method, image comparison method, device, and medium proposed in this application acquire a first image and a second image, as well as the label similarity between the first image and the second image; acquire a first color feature, a first depth feature, and a first semantic feature of the first image, and acquire a second color feature, a second depth feature, and a second semantic feature of the second image; input the first color feature, the first depth feature, and the first semantic feature into a feature fusion sub-model to output a first fused feature, and input the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model to output a second fused feature; input the first fused feature, the first depth feature, and the first semantic feature into a background perception sub-model to output... The first image feature is extracted, and the second fusion feature, the second deep feature, and the second semantic feature are input into the background perception sub-model to output the second image feature. The first image feature is input into the background feature extraction sub-model to output the first background feature, and the second image feature is input into the background feature extraction sub-model to output the second background feature. The predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature. The image comparison model is iteratively trained based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes the trained feature fusion sub-model, the trained background perception sub-model, and the trained background extraction sub-model.

[0017] The trained image contrast model integrates features from three modalities: color, depth, and semantics. A trained feature fusion sub-model fuses these features, a trained background awareness sub-model outputs image features that highlight the background, and a trained background feature extraction sub-model extracts the background features. By comparing the similarity between the background features of different images, the background similarity between them can be accurately measured. Compared to related technologies that directly cut out the foreground and background and then compare the similarity between the backgrounds of different images, this application achieves more accurate background similarity calculation. Attached Figure Description

[0018] Figure 1 This is a flowchart of the image contrast model training method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the image comparison model provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the image comparison method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the image contrast model training device provided in the embodiments of this application; Figure 5This is a schematic diagram of the structure of the image comparison device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0023] Based on this, embodiments of this application provide an image comparison model training method, an image comparison method, an apparatus, and a medium. The trained image comparison model can accurately determine the background similarity between two images.

[0024] The recommended methods, apparatuses, electronic devices, and storage media provided in this application are specifically described through the following embodiments. First, the recommended methods in the embodiments of this application are described.

[0025] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0026] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0027] The image comparison model training method and image comparison method provided in this application relate to the field of artificial intelligence technology. The image comparison model training method and image comparison method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the image comparison model training method and image comparison method, but is not limited to the above forms.

[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0029] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as image information, is required, the user's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0030] Figure 1 This is an optional flowchart of the image contrast model training method provided in the embodiments of this application. Figure 1 The methods may include, but are not limited to, the following steps: Step 110: Obtain the first image and the second image, as well as the label similarity between the first image and the second image; Step 120: Obtain the first color feature, the first depth feature, and the first semantic feature of the first image; obtain the second color feature, the second depth feature, and the second semantic feature of the second image. Step 130: Input the first color feature, the first depth feature, and the first semantic feature into the feature fusion sub-model and output the first fused feature; input the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model and output the second fused feature. Step 140: Input the first fusion feature, the first depth feature, and the first semantic feature into the background perception sub-model and output the first image feature; input the second fusion feature, the second depth feature, and the second semantic feature into the background perception sub-model and output the second image feature. Step 150: Input the first image features into the background feature extraction sub-model and output the first background features; input the second image features into the background feature extraction sub-model and output the second background features. Step 160: Determine the predicted similarity between the first image and the second image based on the first background feature and the second background feature, and iteratively train the image comparison model based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

[0031] Steps 110 to 160 will be described in detail below.

[0032] In step 110, the first image and the second image, as well as the label similarity between the first image and the second image, are obtained.

[0033] The first image and the second image are sample images, which can form a sample image pair. The first image and the second image have a corresponding label similarity, that is, the similarity between the background of the first image and the background of the second image. This similarity is the training label.

[0034] In step 120, the first color feature, the first depth feature, and the first semantic feature of the first image are obtained, and the second color feature, the second depth feature, and the second semantic feature of the second image are obtained.

[0035] The first image can be input into three feature models for extracting color, depth, and semantic features, respectively, to obtain the first color feature, first depth feature, and first semantic feature of the first image. The second image can be input into the same three feature models to obtain the second color feature, second depth feature, and second semantic feature of the second image.

[0036] In some implementations, obtaining the first color feature, the first depth feature, and the first semantic feature of the first image includes: (1.1) Input the first image into the pre-trained color feature extraction model and output the first color feature; (1.2) Input the first image into the pre-trained deep feature extraction model and output the first deep feature; (1.3) Input the first image into the pre-trained semantic feature extraction model and output the first semantic feature.

[0037] For example, a pre-trained color feature extraction model can use a lightweight Swin-Tiny network as its backbone to extract visual features corresponding to the R, G, and B channels, i.e., the first color features (RGB features). The specific calculation method of the pre-trained color feature extraction model is as follows: .in, Four channels The numbers are 96, 192, 384, and 768 respectively. The primary color characteristic, This is the first image. .

[0038] The pre-trained deep feature extraction model can be a DPT-Hybrid model. The first image is input into this model to generate deep features, where feature adaptation can be performed using 3×3 convolutions. The specific calculation method is as follows: .in For the first depth feature, three 3×3 convolutional layers are used, each followed by ReLU activation, ultimately outputting a 256-dimensional feature map. D represents the first image.

[0039] The pre-trained semantic feature extraction model can be a Mask2Former model. The Mask2Former model obtains pixel-level class probabilities, i.e., the probability that different pixels in the first image are in the background. This probability is then converted into a feature vector, i.e., the first semantic feature, through a multilayer perceptron. The semantic branch obtains pixel-level class probabilities through Mask2Former. ,in This represents the number of semantic categories (150 categories according to the ADE20K standard). It is then converted into first semantic features using an MLP: .in The first semantic feature is denoted by S, which represents the pixel-level semantic probability. The MLP consists of two fully connected layers with a middle dimension of 512 and an output dimension of 256.

[0040] Similarly, the second image can be input into a pre-trained color feature extraction model to output the second color feature; the second image can be input into a pre-trained depth feature extraction model to output the second depth feature; and the second image can be input into a pre-trained semantic feature extraction model to output the second semantic feature.

[0041] In step 130, the first color feature, the first depth feature, and the first semantic feature are input into the feature fusion sub-model to output the first fused feature, and the second color feature, the second depth feature, and the second semantic feature are input into the feature fusion sub-model to output the second fused feature.

[0042] Please combine Figure 2 , Figure 2 This is a schematic diagram of the image comparison model provided in an embodiment of this application. The image comparison model includes a feature fusion sub-model, a background perception sub-model, and a background extraction sub-model. The feature fusion sub-model can achieve the fusion of features from three modalities: color, depth, and semantics.

[0043] In some implementations, the feature fusion sub-model includes a cross-attention module and a residual module. The input of the cross-attention module is features of three modalities: color, depth, and semantics. The output of the cross-attention module is connected to the input of the residual module, and the residual module outputs the final fused features.

[0044] The first color feature, the first depth feature, and the first semantic feature are input into the feature fusion sub-model, and the first fused feature is output, including: (1.1) The dimensions of the first color feature, the first depth feature and the first semantic feature are unified respectively to obtain the first updated color feature, the first updated depth feature and the first updated semantic feature; (1.2) Input the first updated color feature, the first updated depth feature, and the first updated semantic feature into the cross-attention module, and output the fused attention feature; (1.3) Input the fused attention features into the residual module and output the first fused feature.

[0045] First, the dimensions of the first color feature, first depth feature, and first semantic feature are unified, for example, by unifying each modal feature to 256 dimensions, resulting in the first updated color feature, first updated depth feature, and first updated semantic feature. The specific processing method is described in the following formula: .in For learnable weights, For bias, It is the number of channels corresponding to the original features of the modality. For color, For depth, sem is for semantics.

[0046] Then, the first updated color features, the first updated depth features, and the first updated semantic features are input into the cross-attention module, which outputs the fused attention features. The cross-attention module has a cross-attention mechanism, and the specific calculation formula is as follows: .in To integrate attention features, The channel input first updates the color features. The channel input is the first updated depth feature. The channel input is updated with the first semantic feature. T represents the transpose operation. This is the activation function.

[0047] The fused attention features are then input into the residual module, which outputs the first fused feature. The residual module is then combined with LayerNorm to generate the first fused feature. The specific processing method can be represented as follows: .in, To achieve feature fusion, a feature map integrating RGB, depth, and semantic information was created. It is a layer normalization operation used to stabilize the feature distribution and accelerate the convergence of subsequent training. For the first update of color features, This represents the summation of the attention interaction results between the non-RGB modalities (depth, semantics) and the RGB modalities.

[0048] Similarly, for the second image, the second color feature, the second depth feature, and the second semantic feature can be dimension-unified to obtain the second updated color feature, the second updated depth feature, and the second updated semantic feature. The second updated color feature, the second updated depth feature, and the second updated semantic feature are then input into the cross-attention module to output the fused attention feature. The fused attention feature is then input into the residual module to output the second fused feature.

[0049] It should be noted that the first fusion feature can more accurately express the features in the image, such as the correlation features between the foreground and the background, so as to make the background and foreground more accurately distinguished in the future.

[0050] In step 140, the first fusion feature, the first depth feature, and the first semantic feature are input into the background perception sub-model to output the first image feature, and the second fusion feature, the second depth feature, and the second semantic feature are input into the background perception sub-model to output the second image feature.

[0051] In some implementations, the background-aware sub-model includes a first attention module, a second attention module, a gating weight module, and a fusion module. The first attention module is used to input deep features, the second attention module is used to input semantic features, the outputs of the first and second attention modules are connected to the input of the gating weight module, the output of the gating weight module is connected to the input of the fusion module, and the fusion module outputs image features that highlight the background.

[0052] The first fusion feature, the first deep feature, and the first semantic feature are input into the background-aware sub-model, and the first image feature is output, including: (1.1) Input the first depth feature into the first attention module and output the spatial attention weight corresponding to each pixel in the first image; (1.2) Input the first semantic feature into the second attention module and output the channel attention weights corresponding to the first image; (1.3) Input the first fusion feature, spatial attention weight and channel attention weight into the gating weight module, and output the first gating fusion weight; (1.4) Input the first gating fusion weight and the first fusion feature into the fusion module and output the first image feature.

[0053] Specifically, the first depth feature is input into the first attention module, which outputs the spatial attention weight corresponding to each pixel in the first image. The specific processing method of the attention mechanism of the first attention module is as follows: This is a depth-guided spatial attention weight calculation formula. Its function is to distinguish between "foreground (person)" and "background" regions by using the local depth differences in the image, and to generate spatial attention weights used to suppress the foreground and enhance the background. It is the spatial attention weight, that is, the spatial attention weight at pixel (i,j) in the image (the value is 0~1, the closer to 1 represents the background, and the closer to 0 represents the foreground). It is the sigmoid activation function. It is the size of the local neighborhood N(i,j) centered at pixel (i,j), specifically a 5×5 neighborhood. It is the depth feature corresponding to pixel (i,j). It is the absolute value of the depth difference between pixel (i,j) and its neighboring pixel k. It is a learnable temperature parameter (adaptively adjusted during training to control the sensitivity to depth differences).

[0054] The first semantic feature is input into the second attention module, which outputs the channel attention weights corresponding to the first image. For example, the background category probability sum is first obtained, which is the sum of pixel-level probabilities of all "background categories" in the first image, reflecting the intensity of the region belonging to the background, and is represented as: .in This represents the sum of background category probabilities. It is a predefined set of background categories, such as "walls, floors" etc. (categories marked as "background" in semantic segmentation). It represents the pixel-level probability of category c in the semantic segmentation result.

[0055] Then, based on the background category probability and the channel attention weights corresponding to the first image, the calculation method is as follows: .in, The semantic conditional channel attention weights, with a dimension consistent with the number of channels in the feature map (256 dimensions), are used to assign weights to each feature channel. MLP stands for Multilayer Perceptron, and GAP stands for Global Average Pooling. This formula can be applied to increasing the weights of feature channels for background elements such as walls and floors, and decreasing the weights of feature channels for foreground elements such as people.

[0056] Then, the first fusion feature, spatial attention weight, and channel attention weight are input into the gating weight module, and the first gating fusion weight is output. First, the gating weight is determined, and then the first gating fusion weight is calculated using the first gating weight, spatial attention weight, and channel attention weight.

[0057] The specific processing method for the first gating weight is as follows: Where g is the first gating weight, These are learnable weights. It is the first fusion feature. It is the first spatial attention weight. It is the attention weight of the first channel.

[0058] The specific processing method for the first gating fusion weight is as follows: .in, It is the first gating fusion weight, used for foreground suppression and background enhancement.

[0059] Finally, the first gating fusion weight and the first fusion feature are input into the fusion module, and the first image feature is output. The specific calculation method is as follows: .in, It is the first image feature.

[0060] As shown above, by calculating the first gating fusion weight, the features of the foreground region can be suppressed, while the features of the background region can be enhanced, thus highlighting the background in the first image features. This facilitates the subsequent calculation of background similarity.

[0061] Similarly, for the second image, the second depth feature is input into the first attention module, and the spatial attention weight corresponding to each pixel in the second image is output; the second semantic feature is input into the second attention module, and the channel attention weight corresponding to the second image is output; the second fusion feature, spatial attention weight and channel attention weight are input into the gating weight module, and the second gating fusion weight is output; the second gating fusion weight and the second fusion feature are input into the fusion module, and the second image feature is output.

[0062] In step 150, the first image features are input into the background feature extraction sub-model to output the first background features, and the second image features are input into the background feature extraction sub-model to output the second background features.

[0063] In some implementations, the background feature extraction sub-model includes multiple sequentially connected background attention modules. For example, in a four-tiered background attention module, the output of one background attention module is connected to the input of the next background attention module, with the first background attention module taking a first image feature as input and the fourth background attention module outputting a first background feature.

[0064] In some implementations, the first image features are input into the background feature extraction module, and the first background features are output, including: (1.1) Input the first image features into the first background attention module and output the background attention features; (1.2) Input the background attention features into the next background attention module after the first background attention module, and output the updated background attention features; (1.3) Input the background attention features output by the previous background attention module into the next background attention module until the last background attention module outputs the first background feature.

[0065] The first image features are input into the first background attention module, which outputs background attention features. Each background attention module contains window multi-head self-attention (W-MSA), shift window attention (SW-MSA), and feedforward network (FFN). The layers are downsampled through max pooling, and finally the background attention features are output.

[0066] The background attention features are input into the background attention module following the first background attention module, and the updated background attention features are output.

[0067] This process continues, with the background attention features output by the previous background attention module being input into the next background attention module, until the last background attention module outputs the first background feature.

[0068] For each background attention module, the calculation formula for window multi-head self-attention (W-MSA) is as follows: ; .

[0069] in, It is the output of multi-head self-attention in the window, which is the refined feature obtained after performing attention calculation within a local window on the input feature F. It's a splicing operation. It is the output of the i-th attention head. It is the output projection matrix: mapping the concatenated attention head features to the final output dimension. Let F be the projection matrix of the i-th attention head, and let F be the input feature F, which is mapped to query (Q), key (K), and value (V) matrices respectively.

[0070] The core of multi-head self-attention within a window is "capturing feature relationships of different dimensions through multiple attention heads within a local window." It calculates attention within the local window, accurately capturing local detail relationships in the background (such as the texture continuity of walls or the local layout of furniture). Combined with subsequent "Shifted Window Attention (SW-MSA)," it can further break down window boundaries, achieving "local + global" feature relationship capture, making the structure of background features more coherent. In summary, the core of multi-head self-attention within a local scope is to capture fine-grained relationships of features through multiple attention heads, providing refined features with "clear local structure" for the multi-scale feature aggregation module, which is the foundation for global background feature aggregation.

[0071] Shifted Window Attention (SW-MSA) alternates between regular and shifted window configurations between adjacent levels. W-MSA computes attention within a fixed local window (e.g., a 7x7 window) and can only capture feature associations within that window. However, the global structure of the background (e.g., the overall layout of a room) often spans multiple windows. SW-MSA establishes cross-window associations by "shifting and dividing windows," allowing features from adjacent windows to enter the same window.

[0072] The computational processing method for feedforward networks is as follows: .in, and For learnable parameters, , This is the bias term. GELU is the Gaussian error linear unit (activation function). It is the output of the feedforward network, which is the refined feature obtained after performing a nonlinear transformation on the input feature F.

[0073] Downsampling is performed between layers using max pooling, and the corresponding processing method is as follows: .in, For the output background attention features, It is a max pooling operation, and LN is a layer normalization operation. The input features are for the current layer, and FFN is a feedforward network.

[0074] As described above, the core advantage of this multi-scale feature aggregation design—combining a 4-level pyramid, W-MSA / SW-MSA, FFN, and hierarchical downsampling—is to generate background features that contain both local details and global structure from three dimensions: multi-scale, multi-association, and layer-by-layer abstraction. This ultimately generates background features that are "detailed, structurally coherent, and appropriately dimensional," providing high-quality input for subsequent background similarity calculations and improving the model's accuracy and efficiency in complex indoor scenes.

[0075] For the second image, the features of the second image are input into the first background attention module, and the background attention features are output; the background attention features are input into the next background attention module after the first background attention module, and the updated background attention features are output; the background attention features output by the previous background attention module are input into the next background attention module, until the last background attention module outputs the second background features.

[0076] In step 160, the predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature, and the image comparison model is iteratively trained based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

[0077] The process of determining the predicted similarity between the first image and the second image based on the first background feature and the second background feature includes: (1.1) Determine the first cosine distance matrix between different feature points in the first background feature, and determine the second cosine distance matrix between different feature points in the second background feature; (1.2) Determine the similarity of the probability distributions between the first cosine distance matrix and the second cosine distance matrix; (1.3) Determine the background space consistency between the first image and the second image based on the first gating fusion weight and the second gating fusion weight corresponding to the second image; (1.4) Determine the predicted similarity between the first image and the second image based on the probability distribution similarity and background space consistency.

[0078] For example, the Gromov-Wasserstein distance metric is used to measure the similarity of probability distributions, and the specific calculation method is as follows: Probability distribution similarity measures the similarity of the global distribution structure of background features between two images (e.g., the texture distribution of walls, whether the relative positions of furniture are consistent). F1 is the first background feature, F2 is the second background feature, i,j are the feature point indices of F1, and k,l are the feature point indices of F2. T is the transfer matrix. The smaller the GW value, the more similar the global distribution of background features. It is the first cosine distance matrix between different feature points in the first background feature. It is the second cosine distance matrix between different feature points in the second background feature.

[0079] Then, based on the first gated fusion weight and the second gated fusion weight corresponding to the second image, the background space consistency between the first image and the second image is determined. The specific calculation method is as follows: .in, It is about background space consistency. It is the first gating fusion weight. This is the second gating fusion weight. The closer the background spatial consistency is to 1, the higher the spatial overlap of the background regions.

[0080] Finally, based on probability distribution similarity and background spatial consistency, the predicted similarity between the first and second images is determined. The specific calculation method is as follows: .in, It is a weighting coefficient that prioritizes "feature distribution similarity" (in indoor scenes, the structural features of the background are more important than spatial location). The closer Sim is to 1, the more similar the backgrounds of the two images are.

[0081] As can be seen from the above, we first use the GW distance to determine whether the global structure of the background features is similar, then use the IoU to determine whether the spatial location of the background region is consistent, and finally use weighted fusion to obtain a similarity score that is both accurate and robust, which is suitable for the actual needs of indoor scenes.

[0082] Finally, the image contrast model is iteratively trained based on the difference between predicted similarity and label similarity to obtain the trained image contrast model. For example, if the difference between predicted similarity and label similarity is greater than a preset difference value, it indicates that the image contrast model is making inaccurate predictions and needs to be iteratively trained until the difference between predicted similarity and label similarity is no greater than the preset difference value. This indicates that the image contrast model has converged, and the trained image contrast model is obtained. The trained image contrast model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

[0083] As described above, the image comparison model training method, image comparison method, device, and medium proposed in this application acquire a first image and a second image, as well as the label similarity between the first image and the second image; acquire the first color feature, the first depth feature, and the first semantic feature of the first image, and acquire the second color feature, the second depth feature, and the second semantic feature of the second image; input the first color feature, the first depth feature, and the first semantic feature into a feature fusion sub-model to output a first fused feature, and input the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model to output a second fused feature; input the first fused feature, the first depth feature, and the first semantic feature into a background perception sub-model. In the process, the first image feature is output, and the second fusion feature, the second deep feature, and the second semantic feature are input into the background perception sub-model to output the second image feature; the first image feature is input into the background feature extraction sub-model to output the first background feature, and the second image feature is input into the background feature extraction sub-model to output the second background feature; the predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature, and the image comparison model is iteratively trained based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes the trained feature fusion sub-model, the trained background perception sub-model, and the trained background extraction sub-model.

[0084] The trained image contrast model integrates features from three modalities: color, depth, and semantics. A trained feature fusion sub-model fuses these features, a trained background awareness sub-model outputs image features that highlight the background, and a trained background feature extraction sub-model extracts the background features. By comparing the similarity between the background features of different images, the background similarity between them can be accurately measured. Compared to related technologies that directly cut out the foreground and background and then compare the similarity between the backgrounds of different images, this application achieves more accurate background similarity calculation.

[0085] It should be noted that the trained image contrast model can be applied in the financial and medical fields. For example, in the financial field, during facial recognition, the trained image contrast model can be used to simultaneously identify and compare the background, further enhancing the security of the recognition process. In the medical field, the trained image contrast model can be used to compare and identify lesion images of a user at different times, thereby determining the recovery or deterioration of certain treatment areas and increasing the accuracy of diagnosis and treatment.

[0086] To better understand the image contrast model provided in the embodiments of this application, please refer to... Figure 2 , Figure 2This is a schematic diagram of the input scene of the image comparison model provided in the embodiments of this application.

[0087] The process involves first obtaining a first image and a second image, as well as the label similarity between the first image and the second image. Then, the first image is input into a pre-trained color feature extraction model to output a first color feature. The first image is also input into a pre-trained depth feature extraction model to output a first depth feature. Finally, the first image is input into a pre-trained semantic feature extraction model to output a first semantic feature.

[0088] Similarly, the second image can be input into a pre-trained color feature extraction model to output the second color feature; the second image can be input into a pre-trained depth feature extraction model to output the second depth feature; and the second image can be input into a pre-trained semantic feature extraction model to output the second semantic feature.

[0089] Then, the first color feature, the first depth feature, and the first semantic feature are input into the feature fusion sub-model to output the first fused feature. The second color feature, the second depth feature, and the second semantic feature are input into the feature fusion sub-model to output the second fused feature. Then, the first fusion feature, the first depth feature, and the first semantic feature are input into the background perception sub-model to output the first image feature, and the second fusion feature, the second depth feature, and the second semantic feature are input into the background perception sub-model to output the second image feature. Then, the first image features are input into the background feature extraction sub-model, and the first background features are output. The second image features are input into the background feature extraction sub-model, and the second background features are output. Finally, the predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature.

[0090] Please see Figure 3 , Figure 3 This is a schematic flowchart of the image comparison method provided in an embodiment of this application. The image comparison method includes the following steps: Step 210: Obtain the first target image and the second target image, as well as the first target color feature, the first target depth feature and the first target semantic feature of the first target image, and the second target color feature, the second target depth feature and the second target semantic feature of the second target image.

[0091] The first target image is input into a pre-trained color feature extraction model, which outputs the color features of the first target; the first target image is input into a pre-trained depth feature extraction model, which outputs the depth features of the first target; the first target image is input into a pre-trained semantic feature extraction model, which outputs the semantic features of the first target.

[0092] Similarly, the second image can be input into a pre-trained color feature extraction model to output the second color feature; the second image can be input into a pre-trained depth feature extraction model to output the second depth feature; and the second image can be input into a pre-trained semantic feature extraction model to output the second semantic feature.

[0093] Step 220: Input the first target color feature, the first target depth feature, and the first target semantic feature into the trained feature fusion sub-model and output the first target fusion feature. Input the second target color feature, the second target depth feature, and the second target semantic feature into the trained feature fusion sub-model and output the second target fusion feature.

[0094] Step 230: Input the first target fusion feature, the first target depth feature, and the first target semantic feature into the trained background perception sub-model and output the first target image feature. Input the second target fusion feature, the second target depth feature, and the second target semantic feature into the trained background perception sub-model and output the second target image feature.

[0095] Step 240: Input the first target image features into the trained background feature extraction sub-model and output the first target background features; input the second target image features into the trained background feature extraction sub-model and output the second target background features.

[0096] Step 250: Determine the target similarity between the first target image and the second target image based on the background features of the first target and the background features of the second target.

[0097] As can be seen from the above, the trained image comparison model can accurately extract the background features in the first target image and the second target image. Subsequently, based on the background features of the first target and the second target, the target similarity between the first target image and the second target image can be accurately determined.

[0098] This image comparison method can be applied to multiple scenarios, such as the financial sector. It can use two facial verifications to distinguish whether an object is in a real scene, thereby ensuring the security of users' financial assets.

[0099] Please see Figure 4 , Figure 4 This is a schematic diagram of the image contrast model training device provided in this application embodiment. The image contrast model includes a feature fusion sub-model, a background perception sub-model, and a background extraction sub-model. The image contrast model training device 300 includes: Image acquisition unit 310 is used to acquire a first image and a second image, as well as the label similarity between the first image and the second image; The feature acquisition unit 320 is used to acquire the first color feature, the first depth feature and the first semantic feature of the first image, and to acquire the second color feature, the second depth feature and the second semantic feature of the second image; The feature fusion unit 330 is used to input the first color feature, the first depth feature and the first semantic feature into the feature fusion sub-model and output the first fused feature, and to input the second color feature, the second depth feature and the second semantic feature into the feature fusion sub-model and output the second fused feature. Background perception unit 340 is used to input the first fusion feature, the first depth feature and the first semantic feature into the background perception sub-model and output the first image feature, and input the second fusion feature, the second depth feature and the second semantic feature into the background perception sub-model and output the second image feature; Background extraction unit 350 is used to input the first image features into the background feature extraction sub-model and output the first background features, and input the second image features into the background feature extraction sub-model and output the second background features; Background comparison unit 360 is used to determine the predicted similarity between the first image and the second image based on the first background feature and the second background feature, and to iteratively train the image comparison model based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model and a trained background extraction sub-model.

[0100] In some implementations, the feature acquisition unit 320 is used for: The first image is input into a pre-trained color feature extraction model, which outputs the first color feature. The first image is input into a pre-trained deep feature extraction model, which outputs the first deep feature. The first image is input into a pre-trained semantic feature extraction model, which outputs the first semantic feature.

[0101] In some implementations, the feature fusion sub-model includes a cross-attention module and a residual module; the feature fusion unit 330 is used for: The first color feature, the first depth feature, and the first semantic feature are dimensionally unified to obtain the first updated color feature, the first updated depth feature, and the first updated semantic feature. The first updated color features, the first updated depth features, and the first updated semantic features are input into the cross-attention module, and the fused attention features are output. The fused attention features are input into the residual module, and the first fused feature is output.

[0102] In some implementations, the background-aware sub-model includes a first attention module, a second attention module, a gating weight module, and a fusion module; the background-aware unit 340 is used for: The first depth feature is input into the first attention module, and the spatial attention weight corresponding to each pixel in the first image is output. The first semantic feature is input into the second attention module, and the channel attention weights corresponding to the first image are output. The first fusion feature, spatial attention weight, and channel attention weight are input into the gating weight module, and the first gating fusion weight is output. The first gating fusion weight and the first fusion feature are input into the fusion module, and the first image feature is output.

[0103] In some implementations, the background contrast unit 360 is used for: Determine the first cosine distance matrix between different feature points in the first background feature, and determine the second cosine distance matrix between different feature points in the second background feature; Determine the similarity of the probability distributions between the first cosine distance matrix and the second cosine distance matrix; The background space consistency between the first image and the second image is determined based on the first gated fusion weight and the second gated fusion weight corresponding to the second image. Based on probability distribution similarity and background spatial consistency, the predicted similarity between the first image and the second image is determined.

[0104] In some implementations, the background feature extraction sub-model includes multiple sequentially connected background attention modules; the background extraction unit 350 is used for: The first image features are input into the first background attention module, and the background attention features are output. The background attention features are input into the next background attention module after the first background attention module, and the updated background attention features are output. The background attention features output by the previous background attention module are input into the next background attention module, until the last background attention module outputs the first background feature.

[0105] In this embodiment, the image acquisition unit 310 acquires a first image and a second image, as well as the label similarity between the first image and the second image; the feature acquisition unit 320 acquires a first color feature, a first depth feature, and a first semantic feature of the first image, and acquires a second color feature, a second depth feature, and a second semantic feature of the second image; the feature fusion unit 330 inputs the first color feature, the first depth feature, and the first semantic feature into the feature fusion sub-model, outputs a first fused feature, and inputs the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model, outputs a second fused feature; the background perception unit 340 inputs the first fused feature, the first depth feature, and the first semantic feature into the background perception sub-model, outputs a first... Image features are input into the background perception sub-model, and the second fusion feature, the second deep feature, and the second semantic feature are input into the background perception sub-model to output the second image feature. The background extraction unit 350 inputs the first image feature into the background feature extraction sub-model to output the first background feature, and inputs the second image feature into the background feature extraction sub-model to output the second background feature. The background comparison unit 360 determines the predicted similarity between the first image and the second image based on the first background feature and the second background feature, and iteratively trains the image comparison model based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes the trained feature fusion sub-model, the trained background perception sub-model, and the trained background extraction sub-model.

[0106] It acquires a first image and a second image, along with the label similarity between the first and second images; acquires the first color feature, the first depth feature, and the first semantic feature of the first image, and the second color feature, the second depth feature, and the second semantic feature of the second image; inputs the first color feature, the first depth feature, and the first semantic feature into a feature fusion sub-model to output a first fused feature, and inputs the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model to output a second fused feature; inputs the first fused feature, the first depth feature, and the first semantic feature into a background perception sub-model to output a first image feature, and inputs the second fused feature into the background perception sub-model to output a second image feature. The first image feature, the second deep feature, and the second semantic feature are input into the background perception sub-model, which outputs the second image feature. The first image feature is input into the background feature extraction sub-model, which outputs the first background feature. The second image feature is input into the background feature extraction sub-model, which outputs the second background feature. The predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature. The image comparison model is iteratively trained based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes the trained feature fusion sub-model, the trained background perception sub-model, and the trained background extraction sub-model.

[0107] The trained image contrast model integrates features from three modalities: color, depth, and semantics. A trained feature fusion sub-model fuses these features, a trained background awareness sub-model outputs image features that highlight the background, and a trained background feature extraction sub-model extracts the background features. By comparing the similarity between the background features of different images, the background similarity between them can be accurately measured. Compared to related technologies that directly cut out the foreground and background and then compare the similarity between the backgrounds of different images, this application achieves more accurate background similarity calculation.

[0108] The specific implementation of this image contrast model training device is basically the same as the specific implementation of the image contrast model training method described above, and will not be repeated here.

[0109] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of the image comparison device provided in the embodiments of this application.

[0110] The image comparison device 400 includes: The target image acquisition unit 410 is used to acquire a first target image and a second target image, as well as a first target color feature, a first target depth feature and a first target semantic feature of the first target image, and a second target color feature, a second target depth feature and a second target semantic feature of the second target image; The target feature fusion unit 420 is used to input the first target color feature, the first target depth feature and the first target semantic feature into the trained feature fusion sub-model and output the first target fusion feature, and input the second target color feature, the second target depth feature and the second target semantic feature into the trained feature fusion sub-model and output the second target fusion feature. The target background perception unit 430 is used to input the first target fusion feature, the first target depth feature and the first target semantic feature into the trained background perception sub-model and output the first target image feature, and input the second target fusion feature, the second target depth feature and the second target semantic feature into the trained background perception sub-model and output the second target image feature. The target background extraction unit 440 is used to input the first target image features into the trained background feature extraction sub-model and output the first target background features, and input the second target image features into the trained background feature extraction sub-model and output the second target background features. The target background comparison unit 450 is used to determine the target similarity between the first target image and the second target image based on the first target background features and the second target background features.

[0111] The specific implementation of this image comparison device is basically the same as the specific embodiment of the image comparison method described above, and will not be repeated here.

[0112] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image comparison model training method or image comparison method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0113] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the image comparison model training method or image comparison method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0114] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image comparison model training method or image comparison method.

[0115] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] The image comparison model training method, image comparison method, device, and medium provided in this application embodiment acquire a first image and a second image, as well as the label similarity between the first image and the second image; acquire a first color feature, a first depth feature, and a first semantic feature of the first image, and acquire a second color feature, a second depth feature, and a second semantic feature of the second image; input the first color feature, the first depth feature, and the first semantic feature into a feature fusion sub-model to output a first fused feature, and input the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model to output a second fused feature; input the first fused feature, the first depth feature, and the first semantic feature into a background perception sub-model. The system outputs a first image feature and inputs a second fusion feature, a second deep feature, and a second semantic feature into a background perception sub-model to output a second image feature. It then inputs the first image feature into a background feature extraction sub-model to output a first background feature, and inputs the second image feature into the background feature extraction sub-model to output a second background feature. Based on the first and second background features, the system determines the predicted similarity between the first and second images. It then iteratively trains the image comparison model based on the difference between the predicted similarity and the label similarity to obtain a trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

[0117] The trained image contrast model integrates features from three modalities: color, depth, and semantics. A trained feature fusion sub-model fuses these features, a trained background awareness sub-model outputs image features that highlight the background, and a trained background feature extraction sub-model extracts the background features. By comparing the similarity between the background features of different images, the background similarity between them can be accurately measured. Compared to related technologies that directly cut out the foreground and background and then compare the similarity between the backgrounds of different images, this application achieves more accurate background similarity calculation.

[0118] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0119] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0122] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0123] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0125] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training an image contrast model, characterized in that, The image comparison model includes a feature fusion sub-model, a background perception sub-model, and a background extraction sub-model, comprising: Obtain the first image and the second image, as well as the label similarity between the first image and the second image; Obtain the first color feature, the first depth feature, and the first semantic feature of the first image; obtain the second color feature, the second depth feature, and the second semantic feature of the second image. The first color feature, the first depth feature, and the first semantic feature are input into the feature fusion sub-model to output the first fused feature, and the second color feature, the second depth feature, and the second semantic feature are input into the feature fusion sub-model to output the second fused feature; The first fusion feature, the first depth feature, and the first semantic feature are input into the background perception sub-model to output the first image feature, and the second fusion feature, the second depth feature, and the second semantic feature are input into the background perception sub-model to output the second image feature; The first image feature is input into the background feature extraction sub-model, and the first background feature is output. The second image feature is input into the background feature extraction sub-model, and the second background feature is output. The predicted similarity between the first image and the second image is determined based on the first background feature and the second background feature. The image comparison model is then iteratively trained based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

2. The image contrast model training method according to claim 1, characterized in that, The step of obtaining the first color feature, first depth feature, and first semantic feature of the first image includes: The first image is input into a pre-trained color feature extraction model, which outputs the first color feature. The first image is input into a pre-trained deep feature extraction model, which outputs the first deep feature. The first image is input into a pre-trained semantic feature extraction model, which outputs the first semantic feature.

3. The image contrast model training method according to claim 1, characterized in that, The feature fusion sub-model includes a cross-attention module and a residual module; The step of inputting the first color feature, the first depth feature, and the first semantic feature into the feature fusion sub-model and outputting the first fused feature includes: The first color feature, the first depth feature, and the first semantic feature are dimensionally unified to obtain the first updated color feature, the first updated depth feature, and the first updated semantic feature. The first updated color feature, the first updated depth feature, and the first updated semantic feature are input into the cross-attention module, and the fused attention feature is output. The fused attention features are input into the residual module, and the first fused feature is output.

4. The image contrast model training method according to claim 1, characterized in that, The background perception sub-model includes a first attention module, a second attention module, a gating weight module, and a fusion module; The step of inputting the first fused feature, the first deep feature, and the first semantic feature into the background-aware sub-model and outputting the first image feature includes: The first depth feature is input into the first attention module, and the spatial attention weight corresponding to each pixel in the first image is output. The first semantic feature is input into the second attention module, and the channel attention weights corresponding to the first image are output. The first fusion feature, the spatial attention weight, and the channel attention weight are input into the gating weight module, and the first gating fusion weight is output. The first gating fusion weight and the first fusion feature are input into the fusion module, and the first image feature is output.

5. The image contrast model training method according to claim 4, characterized in that, Determining the predicted similarity between the first image and the second image based on the first background feature and the second background feature includes: Determine the first cosine distance matrix between different feature points in the first background feature, and determine the second cosine distance matrix between different feature points in the second background feature; Determine the similarity of the probability distributions between the first cosine distance matrix and the second cosine distance matrix; Based on the first gated fusion weight and the second gated fusion weight corresponding to the second image, the background space consistency between the first image and the second image is determined; Based on the probability distribution similarity and the background space consistency, the predicted similarity between the first image and the second image is determined.

6. The image contrast model training method according to claim 1, characterized in that, The background feature extraction sub-model includes multiple background attention modules connected in sequence; The step of inputting the first image features into the background feature extraction module and outputting the first background features includes: The first image features are input into the first background attention module, and the background attention features are output. The background attention features are input into the next background attention module after the first background attention module, and the updated background attention features are output. The background attention features output by the previous background attention module are input into the next background attention module until the last background attention module outputs the first background feature.

7. An image comparison method, characterized in that, An image comparison model trained based on the image comparison model training method according to any one of claims 1 to 6, wherein the method comprises: Acquire a first target image and a second target image, as well as a first target color feature, a first target depth feature and a first target semantic feature of the first target image, and a second target color feature, a second target depth feature and a second target semantic feature of the second target image; The first target color feature, the first target depth feature, and the first target semantic feature are input into the trained feature fusion sub-model to output the first target fusion feature. The second target color feature, the second target depth feature, and the second target semantic feature are input into the trained feature fusion sub-model to output the second target fusion feature. The first target fusion feature, the first target depth feature, and the first target semantic feature are input into the trained background perception sub-model to output the first target image feature. The second target fusion feature, the second target depth feature, and the second target semantic feature are input into the trained background perception sub-model to output the second target image feature. The first target image features are input into the trained background feature extraction sub-model to output the first target background features, and the second target image features are input into the trained background feature extraction sub-model to output the second target background features. The target similarity between the first target image and the second target image is determined based on the first target background features and the second target background features.

8. An image contrast model training device, characterized in that, The image comparison model includes a feature fusion sub-model, a background perception sub-model, and a background extraction sub-model, comprising: An image acquisition unit is used to acquire a first image and a second image, as well as the label similarity between the first image and the second image; The feature acquisition unit is used to acquire the first color feature, the first depth feature, and the first semantic feature of the first image, and to acquire the second color feature, the second depth feature, and the second semantic feature of the second image; The feature fusion unit is used to input the first color feature, the first depth feature, and the first semantic feature into the feature fusion sub-model and output the first fused feature, and to input the second color feature, the second depth feature, and the second semantic feature into the feature fusion sub-model and output the second fused feature; The background perception unit is used to input the first fusion feature, the first depth feature, and the first semantic feature into the background perception sub-model and output the first image feature, and to input the second fusion feature, the second depth feature, and the second semantic feature into the background perception sub-model and output the second image feature; The background extraction unit is used to input the first image features into the background feature extraction sub-model and output the first background features, and to input the second image features into the background feature extraction sub-model and output the second background features; The background comparison unit is used to determine the predicted similarity between the first image and the second image based on the first background feature and the second background feature, and to iteratively train the image comparison model based on the difference between the predicted similarity and the label similarity to obtain the trained image comparison model. The trained image comparison model includes a trained feature fusion sub-model, a trained background perception sub-model, and a trained background extraction sub-model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the image contrast model training method of any one of claims 1 to 6 or the image contrast method of claim 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the image comparison model training method according to any one of claims 1 to 6 or the image comparison method according to claim 7.

Images