Image query method, electronic device, and computer program product

Abstract

The present disclosure provides an image query method, an electronic device and a computer program product. The method of the present disclosure firstly vectorizes a to-be-queried image through a trained image encoder to obtain a to-be-queried vector, then queries N vectors closest to the to-be-queried vector to obtain corresponding N images as a query result, and the vector distance between the output images of the trained image encoder can reflect the structural similarity of the spatial regions between the corresponding input images. The iterative training step of the image encoder includes: determining the difference between the structural similarities of similar image pairs and dissimilar image pairs to obtain a target parameter, and introducing the target parameter and driving a target distance value to increase with the increase of the target parameter, so as to map the relative difference of the structural similarity to the relative separation degree of the vector distance, and the target distance value is the difference between the vector distance between the similar image pairs and the vector distance between the dissimilar image pairs.

Description

Technical Field

[0001] This disclosure specifically relates to image query methods, electronic devices, readable storage media, and computer program products. Background Technology

[0002] Users sometimes need to perform image searches to find similar images, thereby helping them make decisions or assisting in analysis.

[0003] Currently, common image retrieval methods use metrics such as Graph Edit Distance (GED) to measure images. However, GED has high computational complexity, making it difficult to meet the requirements of low latency and high throughput for large-scale online retrieval. Furthermore, it is susceptible to interference from image style, which may lead to similar images of different styles being classified as completely different images, resulting in poor retrieval performance for images of different styles. Summary of the Invention

[0004] This disclosure provides image querying methods, electronic devices, readable storage media, and computer program products.

[0005] The first aspect of this disclosure proposes an image query method, comprising: in response to receiving a query image, vectorizing the query image using a trained image encoder to obtain a query vector; and querying N vectors that are closest to the query vector from a plurality of existing vectors to obtain N corresponding images as query results, where N≥1, wherein the plurality of existing vectors are obtained by vectorizing a plurality of existing images using the trained image encoder, the plurality of existing images correspond to a plurality of image styles, and the vector distance between the output images of the trained image encoder can reflect the structural similarity of spatial regions between corresponding input images; wherein the iterative training step of the image encoder includes: determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs to obtain a target parameter, wherein the similar sample image pairs and the dissimilar sample image pairs share the same reference sample image; and introducing the target parameter and driving the target distance value to increase as the target parameter increases, thereby mapping the relative difference of the structural similarity to the relative separation degree of the vector distance, wherein the target distance value is the difference between the vector distance between the similar sample image pairs and the vector distance between the dissimilar sample image pairs.

[0006] According to some embodiments of this disclosure, determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs includes: determining a first spatial region intersection-union ratio (CIU) between sample images in a similar sample image pair, and determining a second spatial region CIU between sample images in a dissimilar sample image pair; and using the difference between the first spatial region CIU and the second spatial region CIU as the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs.

[0007] According to some embodiments of this disclosure, the calculation method of the spatial region intersection-union ratio between sample images in a sample image pair includes: determining the intersection and union of the mask image of the first sample image and the mask image of the second sample image, and taking the quotient of the intersection and the union as the spatial region intersection-union ratio, wherein the mask image can represent the spatial structure information of the spatial region in the sample image.

[0008] According to some embodiments of this disclosure, the iterative training step further includes: generating multiple sample images corresponding to multiple image styles, and generating mask images of the multiple sample images; and constructing similar sample image pairs and dissimilar sample image pairs based on the cross-union ratio of the mask images.

[0009] According to some embodiments of this disclosure, introducing the target parameter and driving the target distance value to increase as the target parameter increases includes: determining the difference between the vector distance between the sample reference image and the negative sample image in the dissimilar sample images and the vector distance between the sample reference image and the positive sample image in the similar sample images to obtain the target distance value; and if the sum of the target distance value and the target parameter is greater than zero, then continuing the iterative training of the image encoder.

[0010] According to some embodiments of this disclosure, before determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs, the step of training the image encoder further includes: using multiple image-text pairs as training data, obtaining a first vector of the image in the image-text pair through the image encoder, and obtaining a second vector of the text in the image-text pair through the text encoder; calculating the vector similarity between the multiple images and the multiple texts, wherein the image in the image-text pair is a sample image, and the text in the image-text pair is used to describe the spatial structure of spatial regions in the image; and updating the parameters of the image encoder and the text encoder based on the vector similarity, such that the vector distance between the output images of the image encoder can reflect the structural similarity of spatial regions between the corresponding input images.

[0011] According to some embodiments of this disclosure, updating the parameters of the image encoder and the text encoder based on the vector similarity includes: treating images and text as different types of data items, determining the vector similarity between each data item in the training data and each data item of another type; calculating the matching probability of the data item with the data item of another type in the image-text pair using the vector similarity; and calculating a first loss value corresponding to the image and a second loss value corresponding to the text in the image-text pair using the matching probabilities of the two types of data items, and updating the parameters of the image encoder and the text encoder using the first loss value and the second loss value.

[0012] According to some embodiments of this disclosure, the text in the image-text pair is obtained by parsing the vector data of the image in the image-text pair, and the text includes one or more of the shape, proportion, position, and layout of the overall spatial region.

[0013] A second aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the image query method described in any of the above embodiments.

[0014] A third aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, implements the image query method described in any of the above embodiments.

[0015] This disclosure provides a fourth aspect of a computer program product, the computer program product comprising a computer program, which, when executed by a processor, is used to implement the image query method described in any of the above embodiments. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 The illustration shows an application scenario of the image query method according to some embodiments of the present disclosure.

[0018] Figure 2 A schematic diagram of the overall flow of an image query method M100 according to some embodiments of the present disclosure is shown.

[0019] Figures 3-8 A schematic diagram of the overall flow of the iterative training steps M200 of an image encoder according to some embodiments of the present disclosure is shown.

[0020] Figure 9 This is a schematic block diagram of the structure of an image query device according to one embodiment of the present disclosure.

[0021] Figure 10 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure. Detailed Implementation

[0022] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0023] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0025] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0026] Image retrieval can automatically find images that are semantically or visually similar to a given image from an image database. It enables functions such as product search, medical record retrieval, and property search. For example, a user can upload a floor plan of interest to search for properties with similar layouts. Image retrieval can identify multiple properties with similar floor plans from the property database and provide them to the user for selection.

[0027] The commonly used GED (Graphical Image Difference) metric for determining image similarity represents the minimum editing cost required to transform one image into another; a lower metric value indicates greater similarity. However, as image size increases, the computation time for the GED metric grows exponentially, making it difficult to meet users' demands for responsiveness in real-time retrieval scenarios. Furthermore, since GED calculation requires converting the image into a graph structure, differences in image style can lead to errors in graph structure construction, resulting in low accuracy of the calculated metric value and failing to truly reflect the similarity between images.

[0028] Therefore, this disclosure proposes an image query method.

[0029] Figure 1 The diagram illustrates an application scenario of the image query method according to some embodiments of the present disclosure. In this application scenario, a user terminal 10 and a server 20 may be included. The user terminal 10 is capable of communicating with the server 20 to send and receive data or commands. In this disclosure, both the user terminal 10 and the server 20 each include at least one processor and at least one memory.

[0030] For example, a user can upload an image to be queried to a server 20 through a user terminal 10. The server 20 converts the image to be queried into a retrieval vector through an image encoder, and then finds multiple vectors that are closest in the semantic vector space by performing vector retrieval in the vector library. This identifies and recalls multiple images that are most similar to the image to be queried in terms of the structure and layout of the spatial region in the image, and then feeds them back to the user terminal 10.

[0031] exist Figure 1 The shape and structure of the client 10 and server 20 shown should not be construed as a limitation on the scope of protection of this disclosure. In this disclosure, "client 10" can be different types of electronic devices, such as mobile phones, tablets, laptops, or desktop computers. Furthermore, server 20 can be a physical server or a cloud server; this disclosure does not limit the type of server.

[0032] Figure 2 A schematic diagram illustrating the overall flow of an image query method M100 according to some embodiments of this disclosure is shown. For example... Figure 2 The method shown includes steps S110 and S120. This method can be executed by electronic devices such as mobile phones and computers.

[0033] S110, in response to receiving the image to be queried, the image to be queried is vectorized by a trained image encoder to obtain the vector to be queried.

[0034] S120: Query the N vectors closest to the query vector from among multiple existing vectors, obtaining the corresponding N images as the query results, where N≥1. These multiple existing vectors are obtained by vectorizing multiple existing images using a trained image encoder. These multiple existing images correspond to various image styles. The vector distance between the output images of the trained image encoder reflects the structural similarity of spatial regions between the corresponding input images.

[0035] Taking the real estate sector as an example, the image to be queried can be any floor plan uploaded by the user. Floor plans may contain noise such as webpage borders, advertisements, and multiple images mixed together, and the image style can be arbitrary. Images with different styles differ in visual presentation, such as varying line thickness, different color fills for the same spatial area, and background interference. Image styles can include hand-drawn sketches, CAD line drawings, standard renderings, etc.

[0036] An image encoder is a deep neural network model, such as the image encoder included in the CLIP (Contrastive Language–Image Pretraining) pre-trained model. An image encoder can learn the joint semantic representation space between images and text, thereby enabling tasks such as zero-shot image classification and cross-modal retrieval. Specifically, the image encoder can employ the EVA02-CLIP-B-16 image tower. In this embodiment, the image encoder maps the input image to a feature vector of a fixed dimension (e.g., 512 dimensions) in a feature space (such as a semantic space).

[0037] The existing images consist of a large number of floor plans stored in the business unit's database, encompassing various image styles. The existing vectors are a set of vectors obtained by vectorizing these existing floor plans using a trained image encoder and stored in a vector library. This vector library can be the FAISS (Facebook AI Similarity Search) vector library, capable of quickly finding the most similar vector to a given vector in a large-scale vector dataset using methods such as nearest neighbor search, and is particularly suitable for searching high-dimensional vectors.

[0038] Floor plans typically include multiple spatial areas, such as bedrooms, living rooms, bathrooms, and kitchens. Structural similarity refers to the similarity between images in terms of spatial layout and topological relationships. This includes the degree of matching in terms of the geometry, positional relationships, and connectivity of rooms, walls, doors, and windows in two floor plans, excluding visual style or texture similarity. Higher structural similarity indicates greater similarity in spatial layout and topological relationships.

[0039] The image encoder used for the query is the same image encoder as the image encoder that already exists in the generated vector library. After training, the image encoder has the ability to map images with similar spatial structures to vectors with similar distances, and is not affected by different image styles. The vector distance can be Euclidean distance.

[0040] If floor plan a1 in the business database has the same or very similar spatial structure to the floor plan a0 to be queried, then even if a1 and a0 have different image styles, the vector distance between the existing vector b1 of floor plan a1 (after encoding by the image encoder) and the query vector b0 of floor plan a0 will be 0 or a very small value. By querying the N vectors in the vector database that are closest to the query vector b0, the existing vector b1 can be recalled. This allows us to find the floor plan a1 that is most similar to the query floor plan a0 in terms of the number of rooms, layout, and room connectivity, without recalling pseudo-similar results that are similar in style but have different room layouts and structures.

[0041] Figure 3 A schematic diagram of the overall flow of the iterative training steps M200 of the image encoder according to some embodiments of this disclosure is shown. (See also...) Figure 3 The iterative training step M200 of the image encoder includes the following steps S210 and S220.

[0042] S210, determine the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs to obtain the target parameters. The similar and dissimilar sample image pairs share the same reference sample image.

[0043] S220 introduces a target parameter and drives the target distance value to increase as the target parameter increases, thereby mapping the relative difference in structural similarity to the relative separation degree of vector distance. Here, the target distance value is the difference between the vector distance between dissimilar sample image pairs and the vector distance between similar sample image pairs.

[0044] Similar and dissimilar image pairs can be derived from constructed image triples. An image triple can include a reference image (anchor), a positive image, and a negative image. The reference image serves as a reference.

[0045] A reference image (anchor) and a positive image form a similar image pair, while the reference image (anchor) and a negative image form a corresponding dissimilar image pair. The positive image and the reference image are spatially similar, such as two images of different styles of the same apartment layout. The negative image and the reference image, however, have significantly different spatial structures, such as images of a one-bedroom and a three-bedroom apartment. In other words, the spatial structural similarity between a positive image and the reference image is greater than the spatial structural similarity between a negative image and the reference image.

[0046] The target parameter Δ reflects the degree of difference in structural similarity and can be calculated from the difference in structural similarity between similar and dissimilar pairs. It's understandable that a scaling factor λ can be introduced when calculating the target parameter Δ to scale the difference in structural similarity. For example, the product of the structural similarity difference and the scaling factor λ can be used as the target parameter Δ. A target parameter Δ can be calculated for each image triplet.

[0047] The target distance value D reflects the distance between image vectors and can be obtained from the image vectors in the image triplet. The target distance value D is the distance from the anchor of the reference image to the positive image, minus the distance from the anchor of the reference image to the negative image. The target distance value D can be used to measure whether the image encoder correctly separates the vectors of dissimilar samples and narrows the vectors of similar samples. The vector distance can be Euclidean distance (L2 distance), or, if the vectors have been L2 normalized, equivalent to cosine distance.

[0048] As described above, the larger Δ is, the more "similar" the spatial structures of the reference sample image and the positive sample image are, and the less "dissimilar" the spatial structures of the reference sample image and the negative sample image are, thus reflecting the relative difference in structural similarity. Furthermore, the smaller D is, the more the image encoder "pulls" the vectors of dissimilar images "further apart" and "pulls" the vectors of similar images "closer together," thus reflecting the relative separation of vector distances.

[0049] The driving force D increases with Δ, which refers to the adaptive adjustment of the distance between sample images in the vector space based on the magnitude of differences in their spatial structures during training. If two floor plans have significant differences in spatial structure (Δ is large), their vectors can be separated by a large distance (D must also be large); if the differences in spatial structure are small (Δ is small), their vectors can be brought close together (D must also be small). D is essentially a monotonically increasing function of Δ.

[0050] Therefore, by mapping the relative difference in structural similarity (represented by Δ) to the relative separation degree of vector distance (represented by D), the "distance scale" of the vector space can dynamically respond to the real "structural difference degree" between images. The trained image encoder can not only distinguish between "similar and dissimilar", but also between "slightly dissimilar and completely dissimilar", achieving a more refined and reasonable similarity measurement, which greatly improves the ranking quality and user experience.

[0051] For example, the target parameter Δ can be a function learned during loss calculation in the iterative training process to "map Δ to D". When calculating the loss, the target parameter Δ can be used as an adaptive margin. The margin is a "safe distance" set in ranking learning (such as Triplet Loss) to force the image encoder to increase the distance between positive and negative samples in the feature space. The margin represents the minimum allowed distance difference; for example, Δ = 0.2 means that the distance from the reference image to the positive image needs to be at least 0.2 smaller than the distance from the reference image to the negative image.

[0052] After training the image encoder, the function of "the vector distance between output images can reflect the structural similarity of spatial regions between corresponding input images" can be realized, which can be used for image querying. The recall results will include multiple existing floor plans that are most similar to the floor plan structure of the user input image and are not affected by the image style.

[0053] The image query method proposed according to the embodiments of this disclosure has structural sensitivity in the output vector rather than relying solely on general semantics. This allows the image encoder to complete high-precision similarity matching with only one forward propagation (converting the input image into a fixed-dimensional vector) and vector retrieval during inference. It eliminates the need for complex calculations required in GED computation, as well as multi-stage model calls and post-processing, thereby reducing computational costs and query response time. In scenarios of large-scale real-time online retrieval, it meets users' requirements for timely response.

[0054] Furthermore, when faced with input images of different textures, colors, and styles, the trained image encoder can perform similar image queries and recalls based solely on the spatial structure and topology of the spatial regions in the image. This accurately reflects the similarity between images, avoids being affected by the image style of the input image, and thus recalls images with similar styles but different spatial structures, significantly improving the retrieval accuracy in cross-style scenarios.

[0055] In addition, this implementation also uses dynamic interval sorting for training to perform similarity retrieval. By explicitly mapping the relative difference of structural similarity to the relative separation degree of vector distance, it can not only distinguish whether two images are "similar" or "unsimilar", but also whether two images are "slightly different" or "completely different", thereby improving the accuracy of image retrieval and realizing end-to-end trainable, no human intervention required, and cross-style robust image query.

[0056] Figure 4 A schematic diagram of the overall flow of the iterative training steps M200 of the image encoder according to other embodiments of this disclosure is shown. See also... Figure 4 In step S210, the target parameter is obtained by determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs. Specifically, this may include the following steps S211 and S212. In steps S211 and S212, the spatial region intersection over union (IoU) is used as the core metric to quantify "structural similarity".

[0057] S211, determine the first spatial region crossover ratio between sample images in a similar sample image pair, and determine the second spatial region crossover ratio between sample images in a dissimilar sample image pair.

[0058] S212, the difference between the cross-union ratio of the first spatial region and the cross-union ratio of the second spatial region is used as the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs to obtain the target parameters.

[0059] The Intersection over Union (IoU) ratio can be used as a measure of regional overlap. It is defined as the ratio of the area of ​​intersection to the area of ​​union of two regions. The value of IoU is greater than or equal to 0 and less than or equal to 1. The larger the value of IoU, the higher the degree of overlap between the two regions.

[0060] The spatial region IoU between the anchor image and the positive image is called the first spatial region intersection-union ratio, abbreviated as IoU1. The spatial region IoU between the anchor image and the negative image is called the second spatial region intersection-union ratio, abbreviated as IoU2. The target parameter Δ = IoU1 - IoU2 is used to reflect the relative difference in the degree of structural overlap between the two pairs of images.

[0061] IoU is suitable for evaluating the consistency of semantic segmentation results. It can effectively measure the degree of overlap between two images in terms of functional region layout, and completely ignores visual style (such as color and lines). Compared with manually labeled similarity pairs, IoU can be obtained unsupervised from images, which is extremely low cost. By using the IoU difference as the target parameter, a quantifiable, differentiable, and zero-labeled supervised signal of structural similarity is achieved, enabling the image encoder to automatically learn to distinguish subtle structural differences, greatly improving training efficiency and retrieval accuracy.

[0062] In step S211, the spatial region intersection-union ratio (IUGR) between sample images in a sample image pair can be calculated as follows: determine the intersection and union of the mask images of the first sample image and the second sample image, and use the quotient of the intersection and union as the spatial region IUGR. Here, the mask image represents the spatial structure information of the spatial regions in the sample image. The sample images refer to the reference sample image, the positive sample image, and the negative sample image.

[0063] A semantic mask is a two-dimensional integer matrix of the same size as the original image (sample image). Each pixel value represents the spatial region to which that pixel belongs, and the type of pixel value corresponds to the number of different spatial regions. For example, 0 represents the living room, 1 represents the master bedroom, 2 represents the kitchen, and so on. Understandably, different spatial regions within the same category have different pixel values; for example, the first bedroom and the second bedroom have different pixel values. The mask image can completely strip away visual style, retaining only the geometry and semantic category of controls, thus representing spatial structure information.

[0064] This implementation calculates structural similarity using a mask image. The mask image provides a style-independent and structurally accurate intermediate representation, accurately reconstructing the spatial structural layout and providing a reliable foundation for IoU. This allows IoU calculation to focus on the real spatial layout, greatly improving the accuracy and stability of structural similarity assessment. Furthermore, compared to directly calculating structural similarity on the original image, this implementation is less susceptible to noise and style interference, and it bypasses the reliance on "precise graph structure extraction," avoiding intermediate parsing errors.

[0065] The semantic mask image is converted from the sample image earlier than the execution step S211. Figure 5 A schematic diagram of the overall flow of the iterative training steps M200 of the image encoder according to other embodiments of this disclosure is shown. See also... Figure 5 Before performing step S211, that is, before determining the first spatial region intersection-union ratio between sample images in similar sample image pairs and the second spatial region intersection-union ratio between sample images in dissimilar sample image pairs, the iterative training step M200 may also include steps S201 and S202.

[0066] S201 generates multiple sample images corresponding to various image styles, and generates mask images for these sample images. These mask images are semantic mask images.

[0067] S202, construct similar sample image pairs and dissimilar sample image pairs based on the intersection-union ratio of the generated mask images.

[0068] In step S201, multiple real original images can be obtained from the business's database, such as multiple standard floor plans. For example, the original images can be transformed using CycleGAN (Cycle-Consistent Adversarial Network) to obtain synthetic images of different styles. CycleGAN is an unsupervised image-to-image translation deep learning model that can transform an image from one domain to another while maintaining content consistency, even without paired training data. Alternatively, StyleShot can be used to transform the original images to obtain hand-drawn style synthetic images. StyleShot can generate a new image that meets the requirements of the prompt and has the artistic style of the reference image, based on a reference style image and a text prompt. Thus, the original image and the synthetic image together form a sample image.

[0069] Then, semantic mask images for these sample images are generated. Understandably, synthetic images with different image styles generated from the same original image will have mask images that are highly similar to, or even completely identical to, the original image's mask image.

[0070] In step S202, a reference sample image anchor can be selected from the original images in the sample images. The original images contain realistic structural diversity, such as one-bedroom, two-bedroom, L-shaped, and I-shaped spatial layouts in floor plans. When selecting the reference sample image, all original images can be used as reference sample images, or a portion of the original images can be randomly sampled from all original images and used as reference sample images.

[0071] Then, for each reference sample image, the intersection-over-union (IoU) ratio of the mask image of the reference sample image with the mask images of the remaining sample images is calculated. For example, positive sample images can be determined from sample images with IoU values ​​higher than the upper threshold y1, or the positive sample image with the highest IoU value can be selected, so that the spatial layout of the positive sample image is highly similar to that of the reference sample image. The positive sample image can even be a composite image of the original image (used as the reference sample image).

[0072] Additionally, negative sample images can be identified from those with IoU values ​​below the intermediate threshold y2 and above the lower threshold y3, where y1>y2>y3>0. This ensures that the spatial layout of the negative sample images differs significantly from the baseline sample images, for example, the number of rooms in the apartment layout and their relative positions may differ markedly. By setting thresholds y2 and y3, the negative sample images are made into difficult negative samples with low but not the lowest IoU values, thus enhancing the training effect.

[0073] Thus, multiple image triples are formed by using the reference sample image determined from the original image and the positive and negative sample images determined from the sample images including the original image and the synthetic image, thereby forming multiple similar sample image pairs and multiple dissimilar sample image pairs.

[0074] Before constructing similar and dissimilar sample image pairs based on the cross-union ratio (IoU) of the generated mask images (step S202), it is necessary to first calculate the IoU between each pair of mask images. The IoU can be calculated as follows: For two mask images whose IoU is to be calculated, generate manipulated images of a portion of the two mask images. Manipulated images include one or more of the following: rotated images and mirrored images. Rotated images are obtained by rotating by at least one rotation angle, and mirrored images are obtained by flipping along at least one axis of symmetry. Then, the mask images and the corresponding manipulated images are used as computable images of the mask images. The IoU between the computable images of the first and second mask images is calculated, or the IoU between the computable images of the first and second mask images is calculated. The largest IoU is taken as the IoU between the two mask images.

[0075] The rotation angle can include one or more of 90°, 180°, and 270°, resulting in, for example, three rotated images. The axis of symmetry can be one or more of a horizontal line (0° line), a 45° line, a vertical line (90° line), and a 135° line, resulting in, for example, four mirrored images. These seven processed images are then added to the initial mask image (which is neither rotated nor flipped). Each mask image can correspond to eight images used for intersection-union (IU) calculations, thus enabling image computation.

[0076] When calculating the Intersection over Union (IoU) of the mask images, the first mask image can be compared with each of the eight computable images of the second mask image. The maximum IoU among the eight IoU values ​​is then used as the IoU value between the first and second mask images. The maximum IoU is the result of the optimal alignment among all transformations.

[0077] Alternatively, the intersection-over-union (IoU) ratio of the second mask image with each of the eight computable images of the first mask image can be calculated, and the maximum IoU ratio among the eight obtained IoU ratios can be used as the IoU value between the first and second mask images. Another option is to calculate the IoU ratio of each of the eight computable images of the first mask image with each of the eight computable images of the second mask image, and use the maximum IoU ratio among the 64 obtained IoU ratios as the IoU value between the first and second mask images.

[0078] By rotating / mirroring the mask image when calculating the intersection-union ratio (IoU), and using segmentation, geometric invariance, and IoU to automatically quantify structural similarity, large-scale self-supervised training can be achieved. This avoids underestimating the structural similarity between images due to differences in image orientation, allowing structural similarity measurement to focus on the structural and topological relationships of spatial regions.

[0079] The specific methods for calculating the cross-union ratio (CUP) of two images may include: calculating the CUP of spatial regions of various spatial region types in the two images; and weighting the calculated CUP to obtain the CUP of the two images.

[0080] The spatial area types in a floor plan include functional areas such as living room, bedroom, and kitchen. Each spatial area type can be considered a semantic type. The intersection-union ratio (IUR) is calculated separately for spatial areas of different semantic types, resulting in multiple IUR values. Each value corresponds to a semantic type (spatial area type) in the image (such as a mask image). If multiple spatial areas in an image belong to the same semantic type, these spatial areas are merged and their IUR is calculated together.

[0081] Then, the obtained intersection-union (IU) ratios are weighted and averaged according to weight coefficients to obtain the final IU ratio of the two images (mask images). The weight coefficients for different semantic types can be the same, or they can be positively correlated with the pixel proportion of that semantic type. Pixel proportion refers to the percentage of pixels of a semantic type in the union of the two images. For example, if the bedroom is larger than the bathroom, the weight coefficient for the bedroom can be greater than that for the bathroom. The challenge is to avoid excessive interference from smaller regions on the overall similarity, ensuring that the similarity measurement focuses on the main spatial structure, and that the image encoder does not affect the learning of the layout of larger regions in order to fit the positions of smaller regions.

[0082] For example, the mask image of the sample image can be obtained by parsing the vector data of the sample image. The vector data can represent the positional and topological information of spatial regions in the sample image.

[0083] The vector data of floor plans (such as JSON / CAD format) contains information such as the location and topological relationships of rooms. From the vector data of the floor plan, structured fields such as polygon coordinates, category labels, adjacency relationships (connectivity relationships), and outer contours of each room can be parsed. Then, the corresponding regions are filled with the pixel values ​​of the corresponding categories to obtain the mask image of the floor plan, thereby providing a structural similarity supervision signal for the training of the image encoder without manual annotation.

[0084] Figure 6 A schematic diagram of the overall flow of the iterative training steps M200 of the image encoder according to other embodiments of this disclosure is shown. See also... Figure 6 In step S220, a target parameter is introduced and the target distance value is driven to increase as the target parameter increases. Specifically, this may include steps S221 and S222.

[0085] S221, determine the difference between the vector distance between the reference image and the negative sample image in dissimilar sample images and the vector distance between the reference image and the positive sample image in similar sample images, and obtain the target distance value.

[0086] S222, if the sum of the target distance value and the target parameter is greater than zero, then continue the iterative training of the image encoder.

[0087] The structural similarity of similar sample image pairs can be denoted as sim(a,p), and the structural similarity of dissimilar sample image pairs can be denoted as sim(a,n). Here, sim(·) is the similarity calculation function, and a, p, and n are the anchor image, positive image, and negative image, respectively. The target parameter Δ can then be equal to sim(a,p) - sim(a,n). If a scaling coefficient λ is introduced, it can be Δ = λ × [sim(a,p) - sim(a,n)]. By using the target parameter to create a significant difference in distance, we can avoid small perturbations between similar pairs that could lead to recall and ranking errors, thus making the query results more stable and reliable.

[0088] The vector distance between similar sample image pairs can be denoted as ||f(a)-f(p)||2, and the vector distance between dissimilar sample image pairs can be denoted as ||f(a)-f(n)||2. Here, f(a), f(p), and f(n) are the high-dimensional vectors output by the image encoder from the anchor image, positive image, and negative image, respectively. ||·||2 represents the L2 norm, used to calculate the vector length, such as the Euclidean distance between two vectors. The target distance value D can be ||f(a)-f(p)||2-||f(a)-f(n)||2.

[0089] During image encoder training, forward propagation and loss calculation can be performed. The loss value L can be calculated using the formula: L = max(0, D + Δ). The max function ensures that the loss value is not less than zero.

[0090] If the loss value L=0, it means D+Δ≤0, i.e., Δ≤||f(a)-f(n)||2-||f(a)-f(p)||2, indicating that the distance from the reference image to the positive image is at least Δ less than the distance from the reference image to the negative image. The gradient of backpropagation is zero, and the update amount of the image encoder is 0. This signifies that the image encoder strictly aligns the sorting of feature distances with the sorting of structural similarity. Training is complete when all batches of data have learned the principle that "the more similar the structure, the closer the feature distance." If the loss value L>0, training is incomplete, and the image encoder parameters are updated through backpropagation using the loss value. After iterative training until convergence (L=0), a well-trained image encoder is obtained.

[0091] Figure 7 A schematic diagram of the overall flow of the iterative training steps M200 of the image encoder according to other embodiments of this disclosure is shown. See also... Figure 7 Before determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs (step S210), steps S203 and S204 are executed first. Steps S203 and S204 are used to perform the first stage of training for the image encoder and the text encoder. After completing the first stage of training, the second stage of training for the image encoder begins through steps S210 and S220.

[0092] Both steps S203 and S204 can be performed after generating multiple sample images corresponding to various image styles in step S201.

[0093] S203 uses multiple image-text pairs as training data. An image encoder obtains the first vector of the image in the image-text pair, and a text encoder obtains the second vector of the text in the image-text pair. Vector similarity is then calculated between the multiple images and the multiple texts. Here, the image in the image-text pair is the sample image, and the text in the image-text pair is used to describe the spatial structure of the spatial regions in the image.

[0094] S204 updates the parameters of the image encoder and text encoder based on vector similarity, so that the vector distance between the output images of the image encoder can reflect the structural similarity of the spatial regions between the corresponding input images.

[0095] A text-image pair can include an image and a text. In the text-image pairs used in the first stage of training, the image can come from multiple sample images generated in step S201, and the text can be a correct textual description of the spatial structure in the sample images of the text-image pair. That is, in a unified text-image pair, the image and text are semantically matched.

[0096] Text encoders and image encoders can belong to the same CLIP model. A text encoder can be a Transformer-based language model that encodes input text into high-dimensional vectors. The output vectors of both text and image encoders can be projected into the same normalized semantic space.

[0097] In the first phase of training, the image encoder and text encoder are trained simultaneously on multiple image-text pair datasets through contrastive learning, that is, the similarity matrix between image features and text features is calculated simultaneously. Then, the parameters of the image encoder and text encoder are updated using the similarity matrix.

[0098] Figure 8 A schematic diagram of the overall flow of the iterative training steps M200 of the image encoder according to other embodiments of this disclosure is shown. See also... Figure 8 In step S204, the method of updating the parameters of the image encoder and the text encoder based on vector similarity may specifically include steps S204a, S204b and S204c, which further illustrate how to update the parameters of the image encoder and the text encoder.

[0099] S204a, treating images and text as different types of data items, determines the vector similarity between each data item in the training data and each data item in another class.

[0100] S204b calculates the matching probability of a data item with another type of data item in the same image-text pair using vector similarity.

[0101] S204c calculates the first loss value corresponding to the image and the second loss value corresponding to the text in the image-text pair by the matching probability of the two data items, and updates the parameters of the image encoder and the text encoder by the first loss value and the second loss value.

[0102] During forward propagation, for the same batch of training data (including M image-text pairs, M>1), M first vectors and M second vectors are obtained. The M first vectors form an M×K image feature matrix, where K is the dimension (e.g., 512-dimensional). The M second vectors form an M×K text feature matrix. The image feature matrix and the text feature matrix can be normalized separately (e.g., L2 normalization), and the length of each feature vector in the normalized feature matrix is ​​1.

[0103] Then, the cosine similarity between each image feature (first vector) and each text feature (second vector) is calculated to obtain an M×M similarity matrix V (vector similarity). The element v[i,j] represents the similarity between the i-th image and the j-th text.

[0104] The matching probability between two data items can be calculated using the formula exp(v), where exp(v) is an exponential function with the natural constant e as its base, and v is the vector similarity. The sum of the calculated values ​​of each data item in the other class corresponding to the current data item is obtained. The quotient of the calculated value of one of the other class data items (the data item corresponding to the current data item) in the image-text pair to which the current data item belongs, and this sum, is the matching probability between the current data item and the other class data item in the image-text pair. It can be understood that the sum of the matching probabilities of the same data item with each data item in the other class is 1.

[0105] This is understandable. If the matching probability is 100%, then -log(100%) = 0, indicating no loss. However, if the matching probability is only 10%, then -log(10%) is approximately -2.3, indicating a significant loss.

[0106] The first and second loss values ​​can be calculated using -log(g) and by averaging, where g is the matching probability of one type of data item in the same image-text pair. For example, when calculating the first loss value, g is the matching probability of the text in the same image-text pair. The first loss value is obtained by averaging the matching probability of each image relative to the text in its corresponding image-text pair (a total of M log losses). Similarly, when calculating the second loss value, g is the matching probability of the images in the same image-text pair. The second loss value is obtained by averaging the matching probability of each text relative to the images in its corresponding image-text pair (again, a total of M log losses). The average of the first and second loss values ​​is the overall loss value.

[0107] An optimizer is used to simultaneously update all parameters of both the image encoder and the text encoder using the overall loss value. Training is iterated multiple times until the overall loss converges. At this point, the first phase of training is complete, and the image-text encoder is able to align semantically similar image-text pairs in a shared embedding space.

[0108] The goal of contrastive learning is to maximize the similarity between the first and second vectors of matched image-text pairs, while minimizing the similarity between the first and second vectors of mismatched image-text pairs. Feature representations are learned by bringing positive samples closer together and pushing negative samples further apart, enabling two encoders to understand the semantics of spatial structure in images and establish alignment between structural descriptions and corresponding images. For example, the first vector of a floor plan showing a two-bedroom apartment with the living room in the center has a high similarity to the second vector of the text "The apartment contains two bedrooms, with the living room in the center."

[0109] The first stage of training enables the image encoder to map structurally semantically consistent floor plans and text to similar locations in a high-dimensional vector space, establishing cross-modal alignment capabilities and directly inducing high-level topological relationships from pixel-level differences. The image encoder that has completed the first stage of training can distinguish between images that are "similar" and "dissimilar," while the image encoder that has completed the second stage of training can also distinguish between images that are "slightly dissimilar" and "completely dissimilar."

[0110] Understandably, the purpose of training the text encoder in the first stage of training is to train the image encoder, enabling it to understand the structural semantics of the apartment layout and injecting prior knowledge of the layout into it. This establishes a high-quality semantic foundation for the second stage of training. The text encoder is not used in the second stage of training or in actual image queries.

[0111] This implementation method features two-stage structure convergence, significantly improving the similarity metric of "structure-first" approaches. It achieves a Top-5 accuracy of 98.99% in cross-style scenarios and possesses zero manual annotation and scalable data construction capabilities.

[0112] For example, when generating a mask image of a sample image using vector data of an image, a text description of the sample image can also be generated using the vector data, thereby forming an image-text pair.

[0113] For example, the text in a text-image pair can be obtained by parsing the vector data of the images in the text-image pair. The text may include one or more of the following: the shape, scale, position, and layout of the overall spatial region.

[0114] The text description may include one or more of the following: shape, proportion, location, and layout information of the spatial area in the sample image. For a floor plan, shape includes the approximate shape of the overall outline of the unit, such as rectangular or L-shaped. Proportion includes the ratio of width to depth. Location includes the orientation of each room's centroid relative to the overall centroid of the house. Layout includes the adjacency and connectivity relationships between rooms. For example, the text description of a floor plan might include the following: "This unit is rectangular, with a width-to-depth ratio of 1.2. The master bedroom is located in the northwest, adjacent to the bathroom…".

[0115] The shape information mentioned above can be obtained by acquiring the outer contour polygons of the image. For example, first merge the outermost boundaries of all spatial regions (rooms, walls) to obtain closed polygons, and then classify the polygons by shape. The proportion information mentioned above can be obtained by first calculating the ratio of the larger of the width and height of the smallest bounding rectangle of the image content to the smaller of the two values. The position information mentioned above can be obtained by first calculating the centroid of each spatial region, then calculating the weighted average of the centroids of all spatial regions to obtain the overall centroid, and obtaining the quadrant position of the centroid of each spatial region relative to the overall centroid. The layout information mentioned above can be constructed by whether the spatial regions share boundaries or have connecting channels (such as doors).

[0116] It is understandable that multiple synthetic images generated from an original image, a semantic mask image generated, and a text description generated are all associated with the original image and can be represented by the same ID.

[0117] Additionally, before vectorizing the image, it can be cropped to a specified size. If a bounding box is detected in the image, it can be expanded outward from the center point by a certain distance (e.g., 10% of the width and height) to prevent the loss of edge areas during subsequent cropping. Then, it can be scaled proportionally to the specified size.

[0118] Bounding boxes can be detected using YOLOv5. YOLOv5 (You Only Look Once v5) is a single-stage object detection algorithm that can predict the class, location (bounding box), and confidence score of all objects in an input image.

[0119] The above-mentioned image vectorization includes one or more of the following: vectorizing the image to be queried through step S110, and constructing existing vectors in the vector library by vectorizing existing images. This ensures that the synthetic image generated from the original image is of a specified size, the sample image input during the first stage of training is of a specified size, the existing images in the business database are of a specified size, and the size of the image to be queried before vectorization is of a specified size.

[0120] Based on any of the above embodiments, this disclosure also provides an image query device. Figure 9 This is a schematic block diagram of the structure of an image query device according to one embodiment of the present disclosure. Figure 9 As shown, the image query device includes: a vectorization module 110, a query module 120, a similarity calculation module 210, and a difference mapping module 220. The similarity calculation module 210 and the difference mapping module 220 are used for iterative training of the image encoder.

[0121] The vectorization module 110 is used to vectorize the query image to obtain the query vector by using a trained image encoder in response to receiving the query image.

[0122] The query module 120 is used to find the N vectors closest to the vector to be queried from multiple existing vectors, obtaining the corresponding N images as the query results, where N≥1. The multiple existing vectors are obtained by vectorizing multiple existing images using a trained image encoder. These multiple existing images correspond to various image styles. The vector distance between the output images of the trained image encoder reflects the structural similarity of the spatial regions between the corresponding input images.

[0123] The similarity calculation module 210 is used to determine the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs, thereby obtaining the target parameters. The similar and dissimilar sample image pairs share the same reference sample image.

[0124] The difference mapping module 220 is used to introduce target parameters and drive the target distance value to increase as the target parameters increase, thereby mapping the relative difference in structural similarity to the relative separation degree of vector distance. The target distance value is the difference between the vector distance between dissimilar sample image pairs and the vector distance between similar sample image pairs.

[0125] The aforementioned image query device can be in the form of computer software, and each module of the aforementioned image query device can be implemented through computer software modules. The specific implementation process of the functions and roles of each module in the aforementioned device is detailed in the corresponding steps of the aforementioned method, and will not be repeated here.

[0126] The execution subject of the image query method in the specific embodiments of this disclosure can be an electronic device such as a computer or server.

[0127] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the image query method of any of the embodiments described above.

[0128] Figure 10 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0129] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0130] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0131] The processor 1200 can be a central processing unit (CPU). The processor 1200 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0132] The memory 1300 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions of the computer program in the embodiments of this disclosure. The processor 1200 implements the image query method by running the non-transitory software programs, instructions, and modules stored in the memory 1300.

[0133] The memory 1300 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created by the processor 1200. Furthermore, the memory 1300 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1300 may optionally include memory remotely located relative to the processor 1200, and these remote memories may be connected to the processor 1200 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0135] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this disclosure are performed wholly or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0136] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0137] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0142] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0143] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. An image query method, characterized in that, include: In response to receiving a query image, the query image is vectorized using a trained image encoder to obtain a query vector; as well as The N vectors that are closest to the vector to be queried are found among multiple existing vectors, and the corresponding N images are obtained as the query results, where N≥1. The multiple existing vectors are obtained by vectorizing multiple existing images through the trained image encoder. The multiple existing images correspond to multiple image styles. The vector distance between the output images of the trained image encoder can reflect the structural similarity of the spatial regions between the corresponding input images. The iterative training steps of the image encoder include: The difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs is determined to obtain target parameters, wherein the similar sample image pairs and the dissimilar sample image pairs share the same reference sample image; and The target parameter is introduced and the target distance value is driven to increase as the target parameter increases, thereby mapping the relative difference of the structural similarity to the relative separation degree of the vector distance. The target distance value is the difference between the vector distance between the similar sample image pairs and the vector distance between the dissimilar sample image pairs.

2. The image query method according to claim 1, characterized in that, Determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs includes: Determine the cross-union ratio (CUI) of a first spatial region between sample images in a pair of similar sample images, and determine the cross-union ratio (CUI) of a second spatial region between sample images in a pair of dissimilar sample images; and The difference between the cross-union ratio of the first spatial region and the cross-union ratio of the second spatial region is used as the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs.

3. The image query method according to claim 2, characterized in that, The methods for calculating the spatial region intersection-union ratio between sample images in a sample image pair include: The intersection and union of the mask images of the first sample image and the mask images of the second sample image are determined, and the quotient of the intersection and the union is used as the spatial region intersection-union ratio. The mask image can represent the spatial structure information of the spatial region in the sample image.

4. The image query method according to claim 3, characterized in that, The iterative training step also includes: Generate multiple sample images corresponding to various image styles, and generate mask images for these sample images; and Similar and dissimilar sample image pairs are constructed based on the intersection-union ratio of mask images.

5. The image query method according to claim 1, characterized in that, Introducing the target parameter and driving the target distance value to increase as the target parameter increases includes: The target distance value is obtained by determining the difference between the vector distance between the reference image and the negative sample image in the dissimilar sample images and the vector distance between the reference image and the positive sample image in the similar sample images; and If the sum of the target distance value and the target parameter is greater than zero, then the iterative training of the image encoder continues.

6. The image query method according to claim 1, characterized in that, The step of training the image encoder further includes, before determining the difference between the structural similarity of similar sample image pairs and the structural similarity of dissimilar sample image pairs; Using multiple image-text pairs as training data, the first vector of the image in the image-text pair is obtained by the image encoder, and the second vector of the text in the image-text pair is obtained by the text encoder. The vector similarity between multiple images and multiple texts is calculated. The image in the image-text pair is the sample image, and the text in the image-text pair is used to describe the spatial structure of the spatial region in the image. as well as The parameters of the image encoder and the text encoder are updated based on the vector similarity, so that the vector distance between the output images of the image encoder can reflect the structural similarity of the spatial regions between the corresponding input images.

7. The image query method according to claim 6, characterized in that, Updating the parameters of the image encoder and the text encoder based on the vector similarity includes: Using images and text as different types of data items, determine the vector similarity between each data item in the training data and each data item in another class; The probability of matching between a data item and another type of data item in the same image-text pair is calculated using the vector similarity; and The first loss value corresponding to the image and the second loss value corresponding to the text in the image-text pair are calculated using the matching probabilities of the two data items, and the parameters of the image encoder and the text encoder are updated using the first loss value and the second loss value.

8. The image query method according to claim 6 or 7, characterized in that, The text in the image-text pair is obtained by parsing the vector data of the image in the image-text pair. The text includes one or more of the following: the shape, proportion, position, and layout of the overall spatial region.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the image query method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is used to implement the image query method according to any one of claims 1 to 8.

Images