A similar image retrieval method, device, equipment and storage medium

Abstract

The present application relates to the technical field of image matching, and discloses a similar image retrieval method, device and equipment and storage medium, comprising: inputting a to-be-retrieved image and a target image into a pre-trained image segmentation model, obtaining panoramic segmentation results and high-level semantic features of each pixel; performing weighted average on feature vectors of pixels inside each entity to obtain entity description features; constructing a spatial relationship matrix of entity pairs, and weighting and aggregating feature vectors of each overlapping area in the matrix into a spatial topology feature; performing similarity matching on entity description features of entities of the same category in two images to obtain entity description similarity; then performing similarity calculation on spatial topology features of the same category and the same topological relationship to obtain spatial topology similarity; and finally, weighting and fusing the two similarities. The present application can be applied to medical image similar case retrieval in the medical health field and bill and voucher image comparison in the financial technology field, and while retaining global structure information, significantly reduces feature storage capacity.

Description

Technical Field

[0001] This invention relates to the field of image matching technology, and can be applied to fields such as medical health and financial technology. In particular, it relates to a similar image retrieval method, apparatus, device and storage medium. Background Technology

[0002] Image retrieval technology is playing an increasingly important role in healthcare and fintech. For example, in healthcare, doctors need to retrieve images similar to current cases from massive amounts of medical images (such as CT scans, MRIs, and pathology slides) to aid in diagnosis; in fintech, it is often necessary to compare invoices, contracts, facial images, or scene images to complete identity verification, fraud prevention audits, and document verification. These applications place stringent demands on the accuracy, efficiency, and storage overhead of retrieval.

[0003] Currently, mainstream methods for similar image retrieval can be divided into two categories. The first category extracts global features of the image (such as color histograms and features of fully connected layers in deep neural networks) for direct matching. This type of method is computationally efficient and has low storage overhead, but its drawback is that it ignores local details in the image. In healthcare scenarios, two lung CT images may have similar overall structures (both showing the chest cavity outline, tracheal direction, and other large scenes), but the subtle differences in lesion areas (such as small nodules or ground-glass opacities) can determine whether the image is benign or malignant. In fintech scenarios, two invoice templates may have the same format, but subtle alterations to the seal, signature, or amount field cannot be captured by global features, resulting in a large number of false positives.

[0004] The second type of method is based on key feature points, such as SIFT (Scale-invariant feature transform), SURF (SpeededUpRobustFeatures), or the recently proposed SuperPoint (a self-supervised neural network based on deep learning for image feature point detection and descriptor generation). These algorithms first detect key points in the image, then generate local feature descriptors for each key point, and finally complete image retrieval by matching these local features. This type of method can capture detailed information and is robust to changes such as occlusion and rotation. However, it has two significant drawbacks. First, the number of key points is usually enormous; for example, a megapixel medical pathology image can generate tens of thousands to hundreds of thousands of key points, resulting in extremely high computational costs for feature storage and matching, making it difficult to deploy in practical applications such as medical image databases or financial document databases that require storing all image features. Second, each local feature only describes a very small neighborhood (usually within tens of pixels) around the key point, containing very limited semantic information. In the healthcare field, this may result in the loss of the topological relationship between the lesion area and the surrounding tissues and organs, even though the lesion area may be detected as a key point. In the fintech field, it may be impossible to distinguish the ownership of multiple similar seals on the same document, because the local neighborhood cannot carry complete seal shape and relative position information. Summary of the Invention

[0005] The purpose of this invention is to provide a similar image retrieval method, apparatus, device, and storage medium, aiming to solve the problems of insufficient global feature information, excessive local feature storage and computational burden, and limited semantic representation ability in existing image retrieval methods.

[0006] In a first aspect, embodiments of the present invention provide a similar image retrieval method, including: The system acquires a target image and a search image, and inputs the target image and the search image into a pre-trained image segmentation model to obtain a first segmentation result corresponding to the search image and a second segmentation result corresponding to the target image. The first segmentation result includes a search panorama segmentation result and a search high-level semantic feature, and the second segmentation result includes a target panorama segmentation result and a target high-level semantic feature. The first segmentation result is subjected to intra-entity weighted feature calculation to obtain the entity description feature set to be retrieved, and the first segmentation result is subjected to geometric pair spatial topological feature calculation to obtain the spatial topological feature set to be retrieved. The target entity description feature set is obtained by performing intra-entity weighted feature calculation on the second segmentation result, and the target spatial topological feature set is obtained by performing geometric pair spatial topological feature calculation on the second segmentation result. The similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set is calculated to obtain the entity description similarity. The similarity of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity. The entity description similarity and the spatial topology similarity are weighted and fused to obtain the overall similarity, and the overall similarity is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, it is determined that the image to be retrieved and the target image are successfully matched.

[0007] In a second aspect, embodiments of the present invention provide a similar image retrieval device, comprising: The acquisition module is used to acquire a search image and a target image, and input the search image and the target image into a pre-trained image segmentation model to acquire a first segmentation result corresponding to the search image and a second segmentation result corresponding to the target image; the first segmentation result includes a search panorama segmentation result and a search high-level semantic feature, and the second segmentation result includes a target panorama segmentation result and a target high-level semantic feature; The retrieval calculation module is used to perform intra-entity weighted feature calculation on the first segmentation result to obtain the entity description feature set to be retrieved, and to perform geometric pair spatial topological feature calculation on the first segmentation result to obtain the spatial topological feature set to be retrieved. The target calculation module is used to perform intra-entity weighted feature calculation on the second segmentation result to obtain a target entity description feature set, and to perform geometric pair space topological feature calculation on the second segmentation result to obtain a target space topological feature set; The entity description calculation module is used to calculate the similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set, and obtain the entity description similarity. The spatial topology calculation module is used to calculate the similarity between the spatial topology features of geometric pairs with the same topological relationship type in the spatial topology feature set to be retrieved and the target spatial topology feature set, and to obtain the spatial topology similarity. The weighted fusion module is used to perform weighted fusion of the entity description similarity and the spatial topology similarity to obtain an overall similarity, and compare the overall similarity with a predetermined matching threshold; The determination module is used to determine that the image to be retrieved and the target image are successfully matched when the overall similarity is greater than the predetermined matching threshold.

[0008] Thirdly, embodiments of the present invention provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the similar image retrieval method described in the first aspect.

[0009] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements the similar image retrieval method described in the first aspect.

[0010] This invention discloses a similar image retrieval method, apparatus, device, and storage medium, comprising: acquiring a target image and a image to be retrieved; inputting the target image and the image to be retrieved into a pre-trained image segmentation model to obtain a first segmentation result corresponding to the target image and a second segmentation result corresponding to the target image; the first segmentation result includes a target panoramic segmentation result and a target high-level semantic feature, and the second segmentation result includes a target panoramic segmentation result and a target high-level semantic feature; performing intra-entity weighted feature calculation on the first segmentation result to obtain a target entity description feature set, and performing geometric pair spatial topological feature calculation on the first segmentation result to obtain a target spatial topological feature set; performing intra-entity weighted feature calculation on the second segmentation result to obtain a target entity description feature set. The process involves collecting and calculating geometric pairs of spatial topological features from the second segmentation result to obtain a target spatial topological feature set. Similarity is calculated between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set to obtain entity description similarity. Similarly, geometric pairs of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set are calculated to obtain spatial topological similarity. The entity description similarity and the spatial topological similarity are then weighted and fused to obtain an overall similarity, which is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, the image to be retrieved and the target image are determined to be successfully matched. This invention utilizes panoramic segmentation to obtain semantic entities in an image and generates compact features for each entity through weighted averaging. Entity-level features replace massive amounts of local keypoint features, significantly reducing storage requirements while preserving rich detail information. Furthermore, the use of geometric pairs of spatial topological features to calculate and match spatial topological features between entity pairs enhances the ability to judge the consistency of the internal structural layout of the image. Meanwhile, by combining entity feature similarity and spatial topological similarity for retrieval, entity features become more semantically discriminative. Combined with spatial constraints, this significantly improves retrieval accuracy and robustness in complex scenarios, achieving a superior balance between storage efficiency and representational capability. This invention also provides a similar image retrieval device, a computer-readable storage medium, and a computer device, all possessing the aforementioned beneficial effects, which will not be elaborated upon further here. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a similar image retrieval method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a similar image retrieval method. Figure 3 This is a schematic diagram of the DE-9IM model. Figure 4 This is a schematic block diagram of a similar image retrieval device; Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more of its features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] The similar image retrieval method provided in this embodiment of the invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the image to be retrieved and the target image from the client, and input the image to be retrieved and the target image into a pre-trained image segmentation model to obtain a first segmentation result for the image to be retrieved and a second segmentation result for the target image. The first segmentation result includes the panoramic segmentation result and the high-level semantic features to be retrieved, and the second segmentation result includes the panoramic segmentation result and the high-level semantic features of the target. Intra-entity weighted feature calculation is performed on the first segmentation result to obtain the entity description feature set to be retrieved, and geometric pair spatial topological feature calculation is performed on the first segmentation result to obtain the spatial topological feature set to be retrieved. Intra-entity weighted feature calculation is performed on the second segmentation result to obtain the target entity description feature set, and geometric pair spatial topological feature calculation is performed on the second segmentation result to obtain the target entity description feature set. The segmentation results are used to calculate the spatial topological features of geometric pairs to obtain the target spatial topological feature set. The similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set is calculated to obtain the entity description similarity. The similarity between geometric pairs of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity. The entity description similarity and spatial topological similarity are then weighted and fused to obtain the overall similarity, which is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, the image to be retrieved and the target image are considered to have matched successfully. In this invention, panoramic segmentation is used to obtain semantic entities in the image, and a compact feature for each entity is generated through weighted averaging. Entity-level features replace massive amounts of local keypoint features, significantly reducing storage requirements while preserving rich detail information. Furthermore, the use of geometric pairs of spatial topological features to calculate and match the spatial topological features between entity pairs enhances the ability to judge the consistency of the internal structural layout of the image. Meanwhile, by combining entity feature similarity and spatial topological similarity for retrieval, entity features become more semantically discriminative. Combined with spatial constraints, this significantly improves retrieval accuracy and robustness in complex scenarios, achieving a superior balance between storage efficiency and representation capabilities. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, AR devices, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0018] Please see Figure 2 and Figure 3 This embodiment provides a similar image retrieval method, including: S101: Obtain the image to be retrieved and the target image, and input the image to be retrieved and the target image into a pre-trained image segmentation model to obtain a first segmentation result corresponding to the image to be retrieved and a second segmentation result corresponding to the target image; the first segmentation result includes the panorama segmentation result to be retrieved and the high-level semantic features to be retrieved, and the second segmentation result includes the panorama segmentation result of the target and the high-level semantic features of the target. Specifically, the system acquires the image to be retrieved and the target image from an image database or real-time acquisition device. The image to be retrieved can be a medical CT image slice or a scanned copy of a financial document uploaded by the user, while the target image is an image from a medical record database or document database to be compared. To ensure segmentation consistency, both images are adjusted to the input size required by the pre-trained image segmentation model, for example, by scaling the length and width to a uniform resolution. Subsequently, the adjusted image to be retrieved is fed into the image branch of the X-Decoder pre-trained large model. This model adopts a transformer-based encoder-decoder architecture. The encoder performs multi-level feature extraction on the image, generating multi-scale feature maps; the decoder performs pixel-level classification and instance-level segmentation on the feature maps, outputting an entity category label (such as nineteen categories including buildings, trees, vehicles, and pedestrians, corresponding to organs, lesions, and blood vessels in the medical field, and seals, signature fields, and amount numeric areas in the financial field) and a corresponding high-level semantic feature vector with a dimension of 512 for each pixel. After network inference, the image to be retrieved yields a panoptic segmentation result, where each pixel is assigned an entity category label, and pixels with the same label within the same connected region constitute an entity. Simultaneously, high-level semantic features are obtained, namely a 512-dimensional feature vector corresponding to each pixel. The target image is then input into the same X-Decoder model in the same manner, performing the same feature extraction, segmentation, and decoding operations to output the target panoptic segmentation result and the target high-level semantic features. These two sets of outputs respectively contain the entity category attribution for each pixel in the target image and the 512-dimensional semantic features for each pixel. Thus, the first segmentation result contains pixel-level semantic labels and dense features of the image to be retrieved, while the second segmentation result contains the corresponding information of the target image. These two sets of outputs serve as direct inputs for subsequent entity-level feature calculations and spatial topological feature calculations, providing a structured data foundation for cross-image entity matching and spatial relationship analysis.

[0019] In some embodiments, all parameters of the encoder trained on the source domain natural image of the X-Decoder pre-trained large model are retained as the basis for feature extraction. An additional domain discriminator is added after the decoder. This discriminator receives the pixel-level feature map output from the decoder as input and outputs a domain classification probability to determine whether the input feature comes from the source domain or the target domain. The source domain image and the target domain image are simultaneously fed into the encoder and decoder to obtain their respective pixel-level features, which are then passed to the domain discriminator. A domain discrimination loss (e.g., binary cross-entropy loss) is calculated, and the parameters of the domain discriminator are updated by backpropagation based on this loss, enabling it to more accurately distinguish the feature source. Subsequently, the parameters of the domain discriminator are fixed, and the backpropagation signal of the domain discrimination loss is continued to be passed to the decoder and encoder to update their parameters. The goal of this step is to ensure that the features extracted by the encoder cannot be distinguished by the domain discriminator as coming from the source domain or the target domain; that is, the encoder learns a domain-invariant feature representation. Repeat the two alternating update steps above: first update the domain discriminator, then update the encoder and decoder, until an adversarial equilibrium is reached. At this point, the domain discriminator cannot effectively distinguish the source of features, while the features extracted by the encoder have domain invariance.

[0020] Concurrent with the adversarial training described above, a pixel-level self-supervised rotation prediction auxiliary task is introduced. Multiple image patches are randomly sampled from the target domain image, and each patch is randomly rotated by 0, 90, 180, or 270 degrees. The rotated image patches are then input into the encoder and decoder. A global feature vector (e.g., obtained through global average pooling) is extracted from the decoder's output features. This global feature vector is then input into a rotation prediction head, which outputs a four-class probability distribution corresponding to the four rotation angle categories. A rotation prediction loss (e.g., cross-entropy loss) is calculated and backpropagated to update the parameters of the encoder, decoder, and rotation prediction head. This auxiliary task forces the encoder to learn semantically irrelevant geometric structural information in the target domain image, such as edge orientation and texture arrangement, because only by understanding the image's geometric orientation can the rotation angle be correctly predicted. Finally, the domain discrimination loss and rotation prediction loss are jointly optimized; the total loss is a weighted sum of the domain discrimination loss and rotation prediction loss. Through multiple iterations of training, the encoder and decoder parameters are adapted to the target domain. These parameters can achieve better panoptic segmentation performance on the target domain image while maintaining sensitivity to geometric structure.

[0021] S102: Perform intra-entity weighted feature calculation on the first segmentation result to obtain the entity description feature set to be retrieved, and perform geometric pair spatial topological feature calculation on the first segmentation result to obtain the spatial topological feature set to be retrieved; In this embodiment, the first segmentation result is subjected to intra-entity weighted feature calculation to obtain a feature set describing the entity to be retrieved, and the first segmentation result is subjected to geometric pair spatial topological feature calculation to obtain a spatial topological feature set to be retrieved, including: Extract all entities to be retrieved from the panoramic segmentation results to be retrieved; For each entity to be retrieved, the feature vector corresponding to each pixel in the region covered by the entity to be retrieved is obtained from the high-level semantic features to be retrieved, and the feature vectors of all pixels in the region covered by the entity to be retrieved are used to form a set of points to be retrieved. Calculate the position coordinates of each pixel in the set of points to be retrieved and determine the geometric center point of the entity to be retrieved; Based on the geometric center point, assign a weight to each pixel in the set of points to be searched; The feature vectors of all pixels in the set of points to be retrieved are weighted and averaged according to their respective assigned weights to obtain the entity description features of the entity to be retrieved. The entity description features of all entities to be retrieved are summarized to obtain the entity description feature set. Extract all entity pairs from the panorama segmentation results to be retrieved, with each entity pair containing two different entities to be retrieved; For each entity pair, the overlapping areas between the interior, boundary and exterior of the two entities to be retrieved in the entity pair are identified through the spatial topology relationship model; The feature vectors corresponding to all pixels in each overlapping region are obtained from the high-level semantic features to be retrieved, and a weighted average calculation is performed on all feature vectors in each overlapping region to obtain the region feature vector corresponding to each overlapping region. Arrange all the region feature vectors according to a predetermined row and column order to obtain the spatial topological features of the entity pairs; The spatial topological features of all entity pairs are combined to obtain the set of spatial topological features to be retrieved.

[0022] This embodiment aggregates the feature vectors of all pixels within the entity's coverage area into a weighted average vector, requiring only one feature vector to be stored for each entity. This compresses the number of features in the image from the pixel or keypoint level to the entity level, significantly reducing storage overhead and computational complexity for subsequent matching. Weights are assigned based on the geometric center point, with the weights decreasing linearly as the distance from the pixel to the center increases. This ensures that high-confidence pixels in the central region contribute more to entity features, while edge pixels contribute less, enhancing the feature's ability to represent the core semantics of the entity and suppressing interference from boundary noise and segmentation errors. Compared to directly using regional averaging or random sampling, this weighting strategy preserves the differences in pixel importance within an entity, making the features of the same type of entity more stable and robust across different images. The final generated entity description feature set retains the overall semantic information of the entity while possessing compact dimensionality, providing a highly discriminative and robust feature foundation for subsequent cross-image entity matching.

[0023] Meanwhile, this embodiment constructs features using entity pairs as basic units, accurately capturing the spatial relationships between different entities. This overcomes the shortcomings of traditional global features, which lack spatial structure information, and the inability of local keypoint features to reflect the topological relationships between entities. By distinguishing the overlapping areas of entities' interiors, boundaries, and exteriors through a spatial topological relationship model, it can refine the spatial interaction patterns between entities, making the features more closely match the real spatial layout of the image. Weighted averaging of the feature vectors of each overlapping area and then regularizing them eliminates noise interference and forms a unified topological feature structure. Compared to scattered keypoint features, this approach has lower storage requirements and is more computationally efficient. Combining all entity pair topological features into a feature set provides a stable spatial structure basis for subsequent image matching, effectively reducing mismatches caused by relying solely on semantic features, and significantly improving the accuracy and reliability of similar image retrieval in complex scenes.

[0024] Specifically, after obtaining the panoramic segmentation results and high-level semantic features to be retrieved, all entities to be retrieved are first extracted from the panoramic segmentation results. Each entity consists of a set of spatially connected pixels with the same entity category label. For example, in a road scene, a vehicle entity corresponds to all consecutive pixel regions labeled as vehicles; in an indoor scene, all pixels of a chair constitute one entity.

[0025] During extraction, a connected component analysis algorithm is typically used to traverse the category label of each pixel, clustering adjacent pixels with the same label into the same entity and assigning a unique identifier to each entity. For each extracted entity to be retrieved, the feature vector corresponding to each pixel within the area covered by the entity needs to be obtained from the high-level semantic features. Since the high-level semantic features are perfectly aligned with the panoramic segmentation results in terms of spatial resolution—meaning each pixel location has both a category label and a feature vector—the feature vector at the corresponding location can be directly indexed on the high-level semantic feature map based on the set of pixel coordinates contained in the entity to be retrieved. These feature vectors are then aggregated to form a set of points to be retrieved, where each element is a multi-dimensional feature vector (e.g., 512-dimensional), and the number of feature vectors equals the number of pixels contained in the entity.

[0026] Simultaneously, for subsequent weighted calculations, the position coordinates of each pixel in the point set need to be calculated. Typically, the row and column numbers of the pixels in the image are used as the two-dimensional coordinates. Then, the geometric center point of the entity to be retrieved is determined: the arithmetic mean of the coordinates of all pixels covered by the entity is calculated, i.e., summing all row coordinates and dividing by the total number of pixels to obtain the center row coordinate, and summing all column coordinates and dividing by the total number of pixels to obtain the center column coordinate. This coordinate point is the geometric center point. If the entity shape is irregular, the geometric center point may fall inside the entity or even near the edge, but it still serves as the benchmark for subsequent distance calculations. Through the above steps, each entity to be retrieved obtains its corresponding set of pixel feature points, the coordinates of each pixel, and the overall geometric center, providing complete input for the next step of weight allocation and weighted averaging.

[0027] After obtaining the point set and geometric center point of each entity to be retrieved, a weight is assigned to each pixel in the point set based on the geometric center point. Specifically, the Euclidean distance from each pixel in the point set to the geometric center point is first calculated, and the maximum value among all distances is recorded as the maximum distance of the edge of the entity to be retrieved. The weight of the pixel located at the geometric center point is set to 1, the weight of the pixel located at the edge of the entity (distance equal to the maximum distance) is set to 0.2, and the weight of the remaining pixels is obtained by linear interpolation between 1 and 0.2 according to the ratio of its distance to the center point to the maximum distance, that is, weight = 1 - (distance / maximum distance) * 0.8. For the center pixel with a distance of 0, the weight is 1; for the edge pixel with a distance equal to the maximum distance, the weight is 0.2; and the weight of the pixels at intermediate distances decreases linearly.

[0028] After obtaining the weight of each pixel, it is multiplied by the corresponding 512-dimensional feature vector. The weighted feature vectors of all pixels are summed, and then divided by the sum of the weights of all pixels to obtain the weighted average feature vector of the entity to be retrieved, which is the entity description feature of the entity to be retrieved, maintaining a 512-dimensional dimension. In this feature vector, the contribution of pixels in the central region is dominant, while the contribution of pixels at the edge is suppressed, thereby reducing the influence of segmentation boundary noise. All entities to be retrieved extracted from the image are traversed, and the above weighted averaging process is repeated for each entity to obtain the entity description feature corresponding to each entity. Finally, these entity description features are organized according to the order or category of the entities to be retrieved to form the entity description feature set. Each feature vector in this feature set is accompanied by its corresponding entity category label (such as building, vehicle, etc.) for subsequent matching with entities of the same category in the target image.

[0029] In specific applications of fintech and healthcare, the process of calculating intra-entity weighted features and geometric pair spatial topological features of the image to be retrieved is as follows. Assume the image to be retrieved is a scanned copy of a financial document, containing multiple entities such as seals, signature fields, and amount numerals; or a lung CT image, containing multiple anatomical entities such as lung lobes, nodules, blood vessels, and trachea. First, all entities are extracted from the panoramic segmentation results. Each entity consists of a group of pixels with the same category label and connected positions. For example, in a financial document, the seal area is labeled as a seal entity, and the signature field is labeled as a signature entity; in a lung CT scan, the nodule area is labeled as a lesion entity, and the blood vessel area is labeled as a blood vessel entity. For each extracted entity, a 512-dimensional feature vector corresponding to each pixel within the area covered by that entity is obtained from the high-level semantic features of the image to be retrieved. These vectors are then aggregated into a point set. Simultaneously, the image coordinates of each pixel within the entity are calculated, and the arithmetic mean of all pixel coordinates is taken as the geometric center point of the entity. For example, the geometric center point of a seal entity is roughly located at the center of the seal, while the geometric center point of a lung nodule entity is located in the core region of the nodule. Based on the geometric center point, a weight is assigned to each pixel in the point set: pixels located at the geometric center point have the highest weight, pixels located at the edge of the entity have the lowest weight, and the weight of pixels in the middle decreases linearly according to the Euclidean distance from the pixel to the center point. Thus, in financial instruments, the stroke and texture features of the seal's central region receive higher weights, while blurred edges or noisy areas receive lower weights; in lung CT scans, the density features of the nodule's core region receive higher weights, while transitional areas at the edges receive lower weights. The 512-dimensional feature vectors of all pixels in the point set are then weighted and averaged according to their assigned weights. This involves multiplying each feature vector by its corresponding weight, summing the results, and then dividing by the total weights to obtain an aggregated 512-dimensional vector, which serves as the entity description feature. This feature condenses the core semantic information of the entity while suppressing boundary noise and segmentation errors. The process iterates through all entities in the image to be retrieved, aggregating the entity description features of each entity to form a feature set for the retrieved entity. In fintech, this feature set can be used to match with similar entities in the target document (e.g., seal to seal, signature to signature) to detect whether the document has been tampered with or forged. In healthcare, this feature set can be used to match with similar lesion entities in a case database to help doctors quickly locate similar lesions. Subsequently, to maintain spatial topological relationships, geometric pair spatial topological features need to be calculated for the entity pairs. The specific process is similar to the above, but the target is the overlapping area between entities, which will not be elaborated here. Through the above steps, the entity-level features of the image to be retrieved are represented compactly and robustly, laying the data foundation for subsequent similarity calculations.

[0030] Next, all entity pairs are extracted from the panoramic segmentation results to be retrieved. Each entity pair contains two different entities to be retrieved. For example, in financial document images, entity pairs could be a seal and signature bar, a monetary amount area and a seal, a QR code and a tax number area, etc.; in lung CT images, entity pairs could be a lung nodule and a blood vessel, a nodule and a pleura, a left lobe and a right lobe, a trachea and aorta, etc. For each extracted entity pair, the DE-9IM spatial topology model is used to identify nine overlapping regions between the interior, boundary, and exterior of the two entities to be retrieved (such as the first entity to be retrieved and the second entity to be retrieved). Specifically, the interior pixel set, boundary pixel set, and exterior pixel set of the first entity to be retrieved, and the interior, boundary, and exterior pixel sets of the second entity to be retrieved are extracted respectively. Then, nine sets of intersections are calculated: interior-interior, interior-boundary, interior-exterior, boundary-interior, boundary-boundary, boundary-exterior, exterior-interior, exterior-boundary, and exterior-exterior.

[0031] Of these nine overlapping regions, some may be empty sets, while others may contain several pixels. For non-empty overlapping regions, a 512-dimensional feature vector corresponding to each pixel within the region is obtained from the high-level semantic features to be retrieved, and a weighted average is used to calculate the semantic feature vector of the region. The weighting of the weighted average is similar to that of the intra-entity weighting: with the geometric center of the overlapping region as the reference, the closer the pixel is to the center, the greater the weight, and the smaller the weight of edge pixels, with the weight decreasing linearly with distance. If the overlapping region is empty, its region feature vector is set to zero.

[0032] After the above calculations, each of the nine overlapping regions of each entity pair yields a 512-dimensional region feature vector. These nine region feature vectors are arranged according to the row and column order of the DE-9IM matrix (i.e., first row: interior-interior, interior-boundary, interior-exterior; second row: boundary-interior, boundary-boundary, boundary-exterior; third row: exterior-interior, exterior-boundary, exterior-exterior), and concatenated into a complete 9×512-dimensional feature representation, serving as the spatial topological feature of the entity pair. In financial instruments, the spatial topological features of the seal and signature area can characterize their relative position, degree of overlap, and distribution of surrounding blank areas. Even with slight shifts in the seal position or minor deformations in the signature area, this feature can still capture the continuous changes in topological relationships through semantic vectors, rather than a simple empty or non-empty determination. In lung CT images, the spatial topological features of nodules and blood vessels can describe key anatomical relationships such as whether a nodule is traversed by a blood vessel, whether it is located next to a blood vessel, and its relative distance from the pleura. This has significant clinical value in determining the benign or malignant nature of nodules.

[0033] After traversing all distinct entity pairs in the image to be retrieved, the spatial topological features corresponding to each entity pair are aggregated to form a spatial topological feature set. This feature set, together with the aforementioned entity description feature set, provides rich semantic information for subsequent cross-image similarity calculations. Entity description features are responsible for appearance matching of entities of the same type, while spatial topological features are responsible for verifying the consistency of layout relationships between entities. The combination of the two can significantly improve the accuracy of anti-counterfeiting detection of financial instruments and retrieval of similar cases in medical images.

[0034] S103: Perform intra-entity weighted feature calculation on the second segmentation result to obtain the target entity description feature set, and perform geometric pair space topological feature calculation on the second segmentation result to obtain the target space topological feature set; In this embodiment, the target entity description feature set is obtained by performing intra-entity weighted feature calculation on the second segmentation result, and the target spatial topological feature set is obtained by performing geometric pair spatial topological feature calculation on the second segmentation result, including: Extract all target entities from the target panoramic segmentation results; For each target entity, the feature vector corresponding to each pixel in the area covered by the target entity is obtained from the high-level semantic features of the target, and the feature vectors of all pixels in the area covered by the target entity are used to form a target point set. Calculate the position coordinates of each pixel in the target point set and determine the geometric center point of the target entity; Based on the geometric center point, assign a weight to each pixel in the target point set; The feature vectors of all pixels in the target point set are weighted and averaged according to their respective assigned weights to obtain the entity description features of the target entity. The entity description features of all target entities are summarized to obtain the target entity description feature set; Extract all entity pairs from the target panorama segmentation results, where each entity pair contains two distinct target entities; For each entity pair, the overlapping areas between the interior, boundary and exterior of the two target entities in the entity pair are identified through the spatial topology relationship model; The feature vectors corresponding to all pixels in each overlapping region are obtained from the high-level semantic features of the target, and a weighted average calculation is performed on all feature vectors in each overlapping region to obtain the region feature vector corresponding to each overlapping region. Arrange all the region feature vectors according to a predetermined row and column order to obtain the spatial topological features of the entity pairs; The spatial topological features of all entity pairs are combined to obtain the target spatial topological feature set.

[0035] This embodiment processes each entity in the target panoramic segmentation result independently, extracts the feature vector of the corresponding pixel from high-level semantic features, and then performs a weighted average based on the distance from the pixel to the geometric center point of the entity, assigning decreasing weights to obtain the entity description features. This process not only compresses the representation of the target image from massive pixel-level features to a limited number of entity-level features, significantly reducing storage and computational overhead, but also effectively highlights the core semantic information of the entity through the center weighting strategy, while suppressing the interference of edge noise and segmentation errors, making the features of the same type of entity more stable and robust in different images. Simultaneously, for all different entity pairs in the target image, a spatial topological relationship model is used to locate nine overlapping regions between the interior, boundary, and exterior of the two entities, and the pixel features within each overlapping region are weighted and averaged to generate a region feature vector, ultimately forming a structured spatial topological feature. This method upgrades the traditional symbolic spatial relationship description to a continuous high-dimensional semantic vector expression, which can not only quantify the geometric shape, distribution density, and local texture of the overlapping regions, but also allows spatial topological relationships to be directly compared through feature vector distances, significantly enhancing the ability to discriminate the spatial layout consistency between entities in the target image. In summary, the joint construction of the target entity description feature set and the target spatial topology feature set provides a semantically rich, dimensionally compact, and geometrically robust feature foundation for subsequent cross-image matching of entities of the same category and spatial topology verification.

[0036] Specifically, after obtaining the target panoramic segmentation result and the target high-level semantic features, entity extraction is first performed on the target image. From the target panoramic segmentation result, the entity category label of each pixel is traversed using a connected component analysis algorithm. Pixels with the same label and spatially adjacent pixels are clustered into the same target entity, and each target entity is assigned a unique identifier. After extraction, a set of all target entities in the target image is obtained, and each target entity contains its set of pixel coordinates and category label.

[0037] For each extracted target entity, the feature vector corresponding to each pixel within the area covered by that target entity needs to be obtained from the high-level semantic features of the target. Since the high-level semantic feature map of the target and the target panoptic segmentation result are perfectly aligned in spatial resolution, the feature vector at the corresponding position can be directly indexed on the high-level semantic feature map using the coordinates of each pixel in the target entity. These feature vectors typically have a fixed dimension, such as 512-dimensional. The feature vectors of all pixels within the area covered by the target entity are collected in pixel order or any other order to form a target point set. The size of this point set is equal to the number of pixels contained in the target entity, and each element in the point set is a multi-dimensional feature vector.

[0038] Simultaneously, for subsequent weighted average calculations, it's necessary to calculate the position coordinates of each pixel in the target point set. Typically, the row number (y-coordinate) and column number (x-coordinate) of the pixel in the image are used as the two-dimensional coordinate values. Then, the geometric center point of the target entity is determined: the arithmetic mean of the coordinates of all pixels covered by the target entity is calculated. This is done by summing the row coordinates of all pixels and dividing by the total number of pixels to obtain the center row coordinate, and summing the column coordinates of all pixels and dividing by the total number of pixels to obtain the center column coordinate. This coordinate point is the geometric center point of the target entity. Even if the entity shape is irregular or contains holes, the geometric center point can still be uniquely determined using the centroid formula, serving as the reference point for subsequent distance measurements. Through these steps, each target entity obtains its corresponding set of pixel feature points, the coordinates of each pixel, and the overall geometric center point, preparing the data for the next steps of weight allocation and feature aggregation.

[0039] After obtaining the point set and geometric center point of each target entity, a weight is assigned to each pixel in the point set based on the geometric center point. Specifically, the Euclidean distance from each pixel in the point set to the geometric center point is first calculated, and the maximum value among all distances is recorded as the maximum distance of the target entity's edge. Pixels located at the geometric center point are assigned a weight of 1, pixels located at the entity's edge (distance equal to the maximum distance) are assigned a weight of 0.2, and the weights of the remaining pixels are obtained by linear interpolation between 1 and 0.2, based on the ratio of their distance to the center point to the maximum distance, i.e., weight = 1 - (distance / maximum distance) * 0.8. For the center pixel with a distance of 0, the weight is 1; for the edge pixel with a distance equal to the maximum distance, the weight is 0.2; and the weights of pixels at intermediate distances decrease linearly.

[0040] After obtaining the weight of each pixel, it is multiplied by the corresponding 512-dimensional feature vector. The weighted feature vectors of all pixels are summed, and then divided by the sum of the weights of all pixels to obtain the weighted average feature vector of the target entity, which is the entity description feature of the target entity, maintaining a 512-dimensional dimension. In this feature vector, the contribution of pixels in the central region is dominant, while the contribution of pixels at the edges is suppressed, thereby reducing the influence of segmentation boundary noise. The above weighted averaging process is repeated for each target entity extracted from the target image to obtain the entity description feature corresponding to each target entity. Finally, these entity description features are organized according to the order or category of the target entities to form a target entity description feature set.

[0041] In a specific embodiment, the process of calculating the weighted features within entities of the target image is consistent with that of the image to be retrieved to ensure the symmetry and consistency of subsequent matching. Assume the target image is a voucher from a standard invoice template library in a fintech system, or a CT image from a labeled case library in a healthcare platform. First, all entities are extracted from the target panoramic segmentation results. In financial instruments, these entities may include company seals, legal representative signature fields, amount number boxes, invoice code areas, QR codes, etc.; in medical images, entities may include anatomical structures such as lung lobes, nodules, vascular bundles, pleural lines, and ribs. For each extracted entity, a 512-dimensional feature vector corresponding to each pixel within the area covered by the entity is obtained from the target's high-level semantic features, and these feature vectors are aggregated into a point set. Simultaneously, the image coordinates of each pixel within the entity are calculated, and the geometric center point of the entity is determined by arithmetic averaging. For example, in financial instruments, the geometric center of a seal entity is typically located near the center of the seal pattern, while the center of a signature panel entity is located at the center of the signature strokes. In medical imaging, the center of a lung nodule entity is located in the core density region of the nodule, and the center of a blood vessel entity is located on the central axis of the vessel lumen. Using this geometric center point as a reference, a weight is assigned to each pixel in the point set: pixels located at the geometric center point have the highest weight, pixels located at the edge of the entity have the lowest weight, and the weight of intermediate pixels decreases linearly according to the Euclidean distance from the pixel to the center point. This weighting strategy in fintech allows for higher weighting of the complete pattern at the center of the seal and the main strokes of the signature, while pixels at the edges, due to scanning noise or minor contamination, have lower weighting, thus enhancing feature robustness. In healthcare, it allows the density features of the nodule core region and the continuity features of the blood vessel central axis to dominate, while reducing the pixel contribution from the edges due to partial volume effects or segmentation errors. The 512-dimensional feature vectors of all pixels in the point set are weighted and averaged according to their assigned weights. This involves multiplying each feature vector by its corresponding weight, summing the results, and then dividing by the total weights to obtain an aggregated 512-dimensional vector, which serves as the entity description feature. This feature compactly expresses the core semantic information of the entity while suppressing interference from boundary noise and incomplete segmentation. All entities in the target image are traversed, and the entity description features of each entity are aggregated to form the target entity description feature set. In fintech, this feature set can be used for comparison with the entity description feature set of the document to be retrieved, to verify the authenticity of the document or identify signs of tampering. In healthcare, this feature set can be used to calculate similarity with lesion entities in the case to be retrieved, assisting doctors in quickly finding lesions with similar morphology from historical case databases, improving diagnostic efficiency and accuracy.

[0042] Next, all entity pairs are extracted from the target panoramic segmentation results. Each entity pair contains two different target entities. In financial instruments, entity pairs may include seals and signature fields, amount numeral areas and tax number areas, QR codes and invoice code fields, etc.; in lung CT images, entity pairs may include lung nodules and blood vessels, nodules and pleura, left and right lobes, trachea and aorta, etc. For each extracted entity pair, the DE-9IM spatial topology relationship model is used to identify nine overlapping regions between the interior, boundary, and exterior of the two target entities (such as the first target entity and the second target entity). Specifically, the internal pixel set, boundary pixel set, and exterior pixel set of the first target entity, and the internal, boundary, and exterior pixel sets of the second target entity are extracted respectively. Then, nine sets of intersections are calculated: interior-interior II, interior-boundary BI, interior-exterior EI, boundary-interior IB, boundary-boundary BB, boundary-exterior EB, exterior-interior IE, exterior-boundary BE, and exterior-exterior EE. In financial instruments, seals and signature areas may overlap internally (if the seal is on the signature), borderly (the edges are tangent), or externally (the two are separated and surrounded by blank space); in lung CT scans, nodules and blood vessels may overlap internally (the blood vessel passes through the nodule), borderly (the blood vessel is close to the edge of the nodule), etc.

[0043] For each non-empty overlapping region, a 512-dimensional feature vector corresponding to each pixel within that region is obtained from the target's high-level semantic features. A weighted average is then calculated for all feature vectors within that region to obtain the region feature vector for each overlapping region. The weights of the weighted average are assigned based on the geometric center of the overlapping region; pixels closer to the center receive larger weights, while edge pixels receive smaller weights, with the weights decreasing linearly with distance. If the overlapping region is empty, its region feature vector is set to zero.

[0044] After the above calculations, each of the nine overlapping regions of each entity pair yields a 512-dimensional region feature vector. These region feature vectors are arranged according to the row and column order of the DE-9IM matrix (first row: interior-interior, interior-boundary, interior-exterior; second row: boundary-interior, boundary-boundary, boundary-exterior; third row: exterior-interior, exterior-boundary, exterior-exterior), and concatenated into a complete 9×512-dimensional feature representation, serving as the spatial topological feature of the entity pair. In fintech, the spatial topological features of the seal and signature area in a target document can accurately characterize their relative position and degree of overlap. Even with slight offsets or scaling between the standard template and the document to be retrieved, this feature can still capture continuous spatial relationship changes through semantic vectors, effectively detecting whether the document has been tampered with or forged. In healthcare, the spatial topological features of nodules and blood vessels in target CT images can describe key anatomical relationships such as whether blood vessels traverse the nodule and whether it is closely attached to the pleura. This is of significant reference value for determining the nature of the nodule (e.g., benign or malignant).

[0045] After traversing all distinct entity pairs in the target image, the spatial topological features corresponding to each entity pair are aggregated to form a target spatial topological feature set. This feature set, together with the target entity description feature set, constitutes a complete feature representation of the target image. This provides a similar spatial topological comparison benchmark for subsequent similarity calculations with the image to be retrieved, enabling consistency verification of spatial layout in addition to matching entity appearance between the two images. This significantly improves the accuracy of anti-counterfeiting detection of financial instruments and retrieval of similar medical images.

[0046] S104: Calculate the similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set to obtain the entity description similarity. In this embodiment, the similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set is calculated to obtain the entity description similarity, including: Calculate the cosine distance between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set; Determine whether the cosine distance is less than a first predetermined distance threshold; If the cosine distance is greater than or equal to the first predetermined distance threshold, the match is deemed to have failed. If the cosine distance is less than the first predetermined distance threshold, the match is considered successful. Sum the cosine distances of all successful matches and calculate the mean of all successful cosine distances to obtain the entity description similarity.

[0047] This embodiment avoids cross-category mismatches by restricting matching between entities of the same category, significantly improving the semantic accuracy of matching. Using cosine distance as a similarity metric is simple and efficient to calculate, and insensitive to the magnitude of the feature vector, stably measuring the directional consistency of two entity features. Setting a first predetermined distance threshold for hard decision-making quickly filters out obviously dissimilar entity pairs, reducing subsequent computational burden. For matching pairs that pass the threshold, the cosine distances of all successful matches are aggregated and averaged to obtain an overall entity description similarity score. This score integrates the similarity of all valid entity matches, avoiding random errors that may arise from a single matching pair. Furthermore, this method relies only on entity description features and does not involve spatial topology, providing a flexible decision branch for subsequent selective fusion of spatial information. That is, more complex spatial topology verification can be enabled when the number of entity matches is large, thus achieving a balance between efficiency and accuracy.

[0048] Specifically, when calculating the similarity between the entity description feature set to be retrieved and the target entity description feature set, it is first necessary to calculate the cosine distance between each entity description feature to be retrieved and the entity description features of entities of the same category in the target entity description feature set. For example, in financial bill comparison, the seal entity feature of the bill to be retrieved will have its cosine distance calculated with each seal entity feature of the target bill (standard template); in lung CT image retrieval, the lung nodule entity feature of the case to be retrieved will have its cosine distance calculated with each nodule entity feature in the target case database. The value range of the cosine distance is 0 to 2, and the smaller the value, the more similar the two feature vectors are.

[0049] Next, for each feature describing the target entity to be retrieved, it is determined whether the cosine distance is less than a first predetermined distance threshold. This threshold can be pre-calibrated experimentally according to specific application scenarios. For example, it can be set to 0.3 in fintech for strict anti-counterfeiting, and to 0.4 in healthcare to balance sensitivity and specificity.

[0050] If the cosine distance is greater than or equal to the threshold, it indicates that the semantic features of the two entities differ significantly and are insufficient to constitute a reliable match. Therefore, the entity pair is considered a failed match and skipped, not included in subsequent similarity summaries. If the cosine distance is less than the threshold, the entity pair is considered a successful match, and the cosine distance value is recorded as a similarity indicator for a valid match. For example, in fintech, if the cosine distance between the seal on the document to be retrieved and the seal in the standard template is 0.2, which is less than the threshold of 0.3, they are considered highly similar and a successful match. If the cosine distance is 0.5, the match is considered a failure, possibly indicating that the seal has been tampered with or forged. In healthcare, if the cosine distance between the nodule to be retrieved and the target nodule is 0.25, which is less than the threshold of 0.4, they are considered similar lesions and can be used for auxiliary diagnosis. If the cosine distance is greater than or equal to the threshold, the target is excluded. Through this pairwise judgment, reliable entity matching pairs can be efficiently screened from a large number of candidate targets, providing effective data for subsequent calculation of the mean of entity description similarity.

[0051] In this embodiment, before obtaining the entity description similarity by summarizing all successfully matched cosine distances and calculating the mean of all successfully matched cosine distances, the following steps are included: Extract the geometric center coordinates of the entities corresponding to each pair of successfully matched entity description features in their respective images to form a set of matching point pairs; A spatial transformation matrix is ​​fitted by randomly iterating to select a pair of matching points; Calculate the projection error of the remaining matching point pairs under the spatial transformation matrix; Matching point pairs with projection errors less than or equal to a predetermined error threshold are marked as interior points, and matching point pairs with projection errors greater than the predetermined error threshold are marked as exterior points. Gather all interior points from this iteration and calculate their corresponding counts to obtain the interior point set and the number of interior points; When the preset number of iterations is reached, the set of inliers corresponding to the iteration with the most inliers is retained. Each inlier in the set corresponds to a pair of successfully matched entity description features.

[0052] This embodiment utilizes a random sampling consistency algorithm to effectively eliminate spatially inconsistent outliers in the matching results, retaining only geometrically consistent interior point matching pairs. Specifically, a small number of matching point pairs are randomly selected each time to fit the spatial transformation matrix. Then, the projection error of the remaining matching point pairs under this transformation is evaluated, and the point pairs with smaller errors are marked as interior points. After repeated iterations, the transformation model with the most interior points and its interior point set are selected, thus obtaining the most reliable matching subset. This process does not rely on prior knowledge of the correct spatial transformation parameters and is robust to noise and mismatches. Compared to directly using all matching pairs to calculate the mean, this method can eliminate erroneous matches caused by repeated textures, occlusion, or segmentation errors, significantly improving the accuracy and stability of subsequent entity description similarity calculations. Furthermore, since only spatially consistent interior points are retained for mean calculation, contamination of the mean by erroneous matching pairs is avoided, making the final similarity score more reflective of the true entity correspondence between the two images. In addition, this method is highly versatile and applicable to any image matching scenario requiring spatial consistency verification.

[0053] Specifically, after obtaining initially matched entity description feature pairs, it is necessary to further filter out outliers with inconsistent spatial locations to ensure the reliability of subsequent average similarity values. Taking financial bill comparison as an example, assume that the bill to be retrieved and the target standard template have already been filtered out with several successfully matched entity pairs through the cosine distance threshold, such as seal to seal, signature field to signature field, and amount number area to amount number area.

[0054] First, the geometric center coordinates of the entities corresponding to each successfully matched entity description feature are extracted in their respective images, forming a set of matching point pairs. In financial instruments, these geometric center coordinates can reflect the positions of the seal center, signature bar centroid, and digital area center; in lung CT images, they can correspond to the center of lung nodules, vascular branch points, and pleural attachment points.

[0055] Subsequently, a matching pair of points is randomly selected (typically a minimum number of pairs should be chosen, such as 2 pairs for similarity transformation or 3 pairs for affine transformation) to fit a spatial transformation matrix. This matrix describes the geometric mapping between the image to be retrieved and the target image. For example, in fintech, this transformation matrix can express the translation, rotation, or slight scaling of a document during scanning; in healthcare, it can express the spatial displacement of lung lobes or nodules caused by changes in patient positioning.

[0056] Then, the projection error of the remaining matching point pairs under the spatial transformation matrix is ​​calculated, which is the Euclidean distance between the transformed position of each point in the image to be retrieved and the corresponding position in the actual target image. A predetermined error threshold is set, for example, 5 pixels in document comparison and 3 millimeters in CT images. Matching point pairs with projection errors less than or equal to the threshold are marked as inliers, considered to conform to the current assumed spatial transformation; matching point pairs with projection errors greater than the threshold are marked as outliers, considered to be mismatches. All inliers in this iteration are aggregated, and their number and corresponding point pair set are recorded. The above process of random selection, fitting transformation, error calculation, and marking inliers is repeated until the preset number of iterations (e.g., 500 or 1000) is reached. The number of inliers may vary in each iteration.

[0057] Ultimately, the set of inliers corresponding to the iteration with the largest number of inliers is retained. Each inlier in this set corresponds to a pair of successfully matched entity description features with consistent spatial location. In fintech, this process can eliminate erroneous matches caused by folded documents, misaligned seals, or scanning distortions. For example, a mismatch between the upper left corner seal on the document being retrieved and the lower right corner seal on the target document would be identified as an outlier and eliminated by the Random Sample Consensus Algorithm (RANSAC). In healthcare, it can eliminate erroneous matches caused by similar anatomical structures with vastly different spatial locations, such as a mismatch between a right lung nodule and a left lung nodule. Only the inliers retained after RANSAC filtering are used to calculate the mean of the cosine distances of all successfully matched inliers, resulting in a more robust and accurate entity description similarity.

[0058] After filtering out outliers using the RANSAC algorithm, a set of successfully matched entity description feature pairs consisting of inliers is obtained. For example, in financial bill comparison, the inlier set may contain entity matching pairs such as seals, signature fields, and amount numeral areas that are spatially consistent between the bill to be retrieved and the standard template; in lung CT image retrieval, the inlier set may contain entity matching pairs such as lung nodules, vascular branches, and pleural attachment points that are spatially consistent between the case to be retrieved and historical cases. Next, it is necessary to summarize these successfully matched cosine distances and calculate their mean to obtain the entity description similarity. Specifically, the cosine distance value corresponding to each successfully matched entity description feature pair is extracted from the inlier set. These cosine distance values ​​are all determined to be less than a first predetermined distance threshold in the previous steps, representing the similarity of each pair of entities in the feature vector direction. The cosine distance values ​​corresponding to all inliers are summed to obtain the total, and then divided by the total number of inliers to calculate the arithmetic mean. This average value is the entity description similarity. In fintech, if a searchable document and a target standard template exhibit a small cosine distance (i.e., high similarity) across multiple entities such as seals and signature fields, and pass spatial consistency verification, the mean of entity description similarity will be close to 0, indicating high similarity and the document's authenticity. Conversely, a large mean suggests potential tampering or forgery. In healthcare, the mean of entity description similarity quantifies the overall similarity between a searchable lesion and lesions in a historical case database, assisting doctors in quickly locating morphologically similar lesions and providing a diagnostic reference. This mean integrates similarity information from all valid entity matches, avoiding the randomness of single entity matching, exhibiting strong robustness and representativeness, and laying the data foundation for subsequent optional fusion spatial topological similarity calculations.

[0059] In this embodiment, after summarizing all successfully matched cosine distances and calculating the mean of all successfully matched cosine distances to obtain the entity description similarity, the following steps are taken: Count the number of successful matches to obtain the total number of matches; If the number of matches is 0, it is determined that the image to be retrieved and the target image have failed to match; If the number of matches is 1, then determine whether the entity description similarity is greater than or equal to the second predetermined distance threshold. If the entity description similarity is greater than or equal to the second predetermined distance threshold, the image to be retrieved and the target image are determined to be unmatched. If the entity description similarity is less than the second predetermined distance threshold, then the image to be retrieved and the target image are determined to be a successful match. If the number of matches is greater than 1, the similarity of the spatial topological features of geometric pairs with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity.

[0060] This embodiment adaptively selects the subsequent processing path based on the number of successfully matched entity pairs (i.e., the number of inliers), achieving a balance between computational efficiency and matching accuracy. When the number of matches is 0, the matching is directly determined to be unsuccessful, avoiding invalid spatial topology calculations for unmatched images and saving computational resources. When the number of matches is 1, a final determination can be made solely by comparing the entity description similarity with a second predetermined distance threshold, without initiating complex spatial topology feature calculations, as a single entity pair cannot constitute a statistically significant spatial relationship. In this case, rapid decision-making is both reasonable and efficient. Spatial topology similarity calculation is introduced only when the number of matches is greater than 1, because multiple entity pairs can provide rich spatial layout information. By fusing the similarity of the entities themselves and the similarity of the spatial relationships between entities, the overall matching degree of the two images can be judged more comprehensively and robustly. This hierarchical decision-making mechanism avoids performing computationally intensive spatial topology matching in all cases, while ensuring that spatial information is fully utilized to improve accuracy in multi-entity matching scenarios, demonstrating good practicality and scalability.

[0061] Specifically, after obtaining the entity description similarity, branching decisions need to be made based on the number of successfully matched entity pairs (i.e., the number of inliers) to determine the subsequent processing path. First, the number of successfully matched entity pairs retained after RANSAC filtering out outliers is counted and denoted as the matching count. For example, in financial bill comparison, the matching count can represent the number of entities such as seals, signature fields, and amount numeral areas that pass the spatial consistency test between the bill to be retrieved and the standard template; in lung CT image retrieval, the matching count can represent the number of entities such as lung nodules, vascular branches, and pleural attachment points that are spatially consistent between the case to be retrieved and historical cases.

[0062] If the number of matches is 0, it means that there is no spatially consistent entity correspondence between the image to be retrieved and the target image, and the two images are directly judged as failing to match. In fintech, this means that the document to be retrieved and the standard template have no similarity in core entities, and may belong to completely different document types or have been severely tampered with; in healthcare, it means that the lesion area of ​​the case to be retrieved and the case to be targeted do not have a consistent anatomical correspondence, and the target should be excluded.

[0063] If the number of matches is 1, only one pair of entities matches successfully. In this case, the spatial topological relationship between multiple entities cannot be used for verification, so the judgment is based solely on the similarity of entity descriptions. Specifically, the similarity of entity descriptions (i.e., the mean of the cosine distances of all successfully matched entities) is judged to be greater than or equal to a second predetermined distance threshold. This threshold can be set according to the application scenario. For example, it is set more strictly (e.g., 0.25) in anti-counterfeiting of financial instruments, and more leniently (e.g., 0.35) in medical image-assisted diagnosis. If the similarity of entity descriptions is greater than or equal to this threshold, it indicates that the only matching entity pair has a large difference in feature vectors, which is insufficient to be considered similar, and the match is judged to have failed. Conversely, if the similarity of entity descriptions is less than this threshold, it indicates that the entity pair is highly similar, and since there is only one entity pair, the spatial layout cannot be further verified, and the match is directly judged to have succeeded. In fintech, this situation may occur in the comparison of a single-seal document with a single seal in a standard template; in healthcare, it may occur in the comparison of an isolated pulmonary nodule with an isolated nodule in a case database that has a similar morphology.

[0064] If the number of matches is greater than 1, it indicates that there are multiple spatially consistent entity correspondences between the two images. In this case, entity description similarity alone is insufficient to comprehensively assess image similarity, and spatial topological information needs to be introduced. Therefore, the next step is to calculate the similarity of the spatial topological features of geometric pairs with the same topological relationship type in the target spatial topological feature set and the search target spatial topological feature set, obtaining the spatial topological similarity. This similarity quantifies the consistency of the spatial layout between entity pairs in the two images. For example, in financial documents, whether the relative position, angle, and degree of overlap between the seal and the signature bar match; in lung CT scans, whether the traversal relationship between nodules and blood vessels, and the adjacency distance between nodules and the pleura are consistent. By fusing entity description similarity and spatial topological similarity, a more comprehensive and robust matching judgment is finally made. This hierarchical decision-making mechanism avoids invalid calculation of spatial topology when the number of matches is insufficient, and ensures that spatial information is fully utilized to improve accuracy in multi-entity matching.

[0065] S105: Calculate the similarity between the geometric pairs of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set to obtain the spatial topological similarity. In this embodiment, the similarity of geometric pairs with the same topological relationship type within the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity, which includes: Extract the spatial topological features of each geometric pair from the set of spatial topological features to be retrieved, along with their corresponding first entity category label, second entity category label, and topological relation type. Extract the spatial topological features of each geometric pair from the target space topological feature set, along with its corresponding first entity category label, second entity category label, and topological relation type; Filter out geometric pairs of spatial topological feature pairs in the spatial topological feature set to be retrieved that have the same first entity category label, the same second entity category label, and the same topological relationship type as those in the target spatial topological feature set. For each geometric pair of spatial topological features, the cosine distance between the region feature vectors corresponding to the two geometric pairs of spatial topological features is calculated in row and column order to obtain multiple similarity values. The spatial topological similarity is obtained by summing all the similarity values ​​calculated for all geometric pairs and spatial topological feature pairs, and then dividing by the total number of similarity values.

[0066] This embodiment employs strict category and topological relationship constraints to ensure that only semantically and spatially comparable entity pairs participate in similarity calculations, avoiding mismatches between cross-category or different topological relationships. Specifically, geometric pairs with identical first entity category, second entity category, and topological relationship type are selected, guaranteeing the homogeneity of spatial relationships between entity pairs in the searched image and the target image, thus giving subsequent feature vector comparisons clear physical meaning. Instead of simply concatenating the entire 9×512-dimensional feature set, the cosine distance of the nine corresponding region feature vectors in the spatial topological features of each geometric pair is calculated in row and column order, preserving the independent similarity contribution of each overlapping region and facilitating the location of differences. Finally, all similarity values ​​are averaged to obtain a global spatial topological similarity score, which integrates the spatial consistency of all valid entity pairs, exhibiting strong robustness and representativeness. This method upgrades the DE-9IM model from discrete symbolic comparison to a continuous vector distance metric, capturing deeper information such as the geometric shape and distribution density of overlapping regions, significantly improving the ability to discriminate the similarity of image spatial layout.

[0067] Specifically, each geometric pair of spatial topological features is extracted from the set of topological features to be retrieved, and the corresponding first entity category label (e.g., a seal), second entity category label (e.g., a signature bar), and topological relationship type (e.g., disjoint, overlapping, or containing) are simultaneously read. Similarly, each geometric pair of spatial topological features and its corresponding first entity category label, second entity category label, and topological relationship type are extracted from the set of topological features to be retrieved. In lung CT image retrieval, the first entity category label can be a lung nodule, the second entity category label can be a blood vessel, and the topological relationship type can be a blood vessel passing through a nodule or a nodule adhering to the pleura.

[0068] Next, all features in the searchable spatial topological feature set are compared one by one with the features in the target spatial topological feature set. Only feature pairs with the same first entity category label, the same second entity category label, and identical topological relationship type are retained. For example, in financial instruments, only the seal-signature pair of the searched instrument (topologically disjoint) and the seal-signature pair of the target instrument (topologically disjoint) can constitute a valid geometric spatial topological feature pair; if the topological relationships are different (one is disjoint, the other is overlapping), they are not matched. In medical images, only the nodule-vessel pair of the searched case (topologically, the vessel passes through the nodule) and the nodule-vessel pair of the target case (also, the vessel passes through the nodule) will be selected.

[0069] For each selected pair of geometric spatial topological features, nine region feature vectors are extracted from each of the two geometric spatial topological features. These nine region feature vectors are arranged in the row and column order of the DE-9IM matrix, corresponding sequentially to interior-interior, interior-boundary, interior-exterior, boundary-interior, boundary-boundary, boundary-exterior, exterior-interior, exterior-boundary, and exterior-exterior. The cosine distance between the first region feature vector of the feature to be retrieved and the first region feature vector of the target feature is calculated to obtain the first similarity value; then the cosine distance between the second region feature vector and the second region feature vector is calculated to obtain the second similarity value; and so on, until all nine region feature vectors have been calculated, resulting in a total of nine similarity values. For example, in financial instruments, if the feature vector of the boundary-internal region of the seal-signature pair of the instrument to be retrieved is very small and the cosine distance of the corresponding region of the target instrument is very small, it indicates that the two are highly consistent in the overlapping pattern of the seal boundary and the interior of the signature field; in medical images, if the cosine distance of the interior-internal region feature vector of the nodule-vessel pair is very small, it indicates that the density distribution of the overlapping area (i.e., the part through which the blood vessel passes) inside the nodule and the blood vessel is very similar.

[0070] The total similarity score is calculated by summing all the similarity values ​​obtained from the selected geometric pairs and spatial topological feature pairs. Simultaneously, the total number of similarity values ​​involved in the calculation is counted; this total number equals the number of selected feature pairs multiplied by nine. Finally, the total similarity score is divided by the total number of similarity values ​​to obtain the average spatial topological similarity. This average quantifies the overall consistency between the image to be retrieved and the target image in the spatial layout of all comparable entity pairs; the smaller the value, the more similar the spatial topology. In fintech, this score can be used to assess whether the relative positions of multiple entities such as seals, signature fields, and amount boxes in a document are consistent with a standard template, thus aiding in determining the authenticity of the document. In healthcare, this score can be used to compare the spatial relationships between multiple lesions and surrounding anatomical structures in different cases, helping doctors quickly locate lesions with similar spatial topological features, improving the accuracy and efficiency of diagnosis.

[0071] S106: The entity description similarity and the spatial topology similarity are weighted and fused to obtain the overall similarity, and the overall similarity is compared with a predetermined matching threshold; Specifically, after obtaining the entity description feature similarity b and spatial topological similarity t, a weighting term λ is set to finally obtain the overall similarity s: s=λ*b+(1-λ)t Then, a predetermined matching threshold is set to M2, and s is compared with the threshold M2.

[0072] S107: When the overall similarity is greater than the predetermined matching threshold, it is determined that the image to be retrieved and the target image are successfully matched.

[0073] If s is greater than M2, the image to be retrieved is determined to be a successful match with the target image, and a successful match flag is output; if s is less than or equal to M2, the match is determined to be a failure, and a failure flag is output. By combining weighted fusion and threshold comparison, the results of entity-level appearance matching and spatial topology consistency verification are integrated, and the final decision is achieved through a single threshold.

[0074] In some embodiments, the method further includes: after obtaining the set of matching interior points (filtered by RANSAC) of entity description features, calculating the variance σ of the cosine distance of each interior point matching pair. b The smaller the variance, the more consistent the appearance of the entities, and the weight of λ should be increased; conversely, if the variance is large, λ should be decreased.

[0075] Simultaneously, the matching stability of the feature vectors of the nine regions in the spatial topological similarity is calculated, that is, the variance σ of the cosine distance of all entity pairs at the same region location (such as the BI region) is calculated. t The smaller the variance, the more stable the spatial topological relationship, and the weight of (1-λ) should be increased.

[0076] Next, λ is calculated using the formula: λ = σ t / (σ b +σ t +ε), which is the proportion of spatial topological variance to total variance. When spatial topological relationships are highly consistent (σ... t When σ is small, λ automatically decreases and becomes more dependent on spatial topology; when the consistency of entity appearance is strong (σ... b When the size is small, λ automatically increases and becomes more dependent on the physical appearance.

[0077] This embodiment utilizes panoramic segmentation to obtain semantic entities in the image and generates compact features for each entity through weighted averaging. Entity-level features replace massive amounts of local keypoint features, significantly reducing storage requirements while preserving rich detail information. Secondly, it employs geometric pair spatial topological features to calculate and match the spatial topological features between entity pairs, enhancing the ability to judge the consistency of the internal structural layout of the image. Simultaneously, by combining entity feature similarity and spatial topological similarity for retrieval, entity features become more semantically discriminative. Combined with spatial constraints, this significantly improves retrieval accuracy and robustness in complex scenes, achieving an excellent balance between storage efficiency and representational capability.

[0078] Please see Figure 4 This embodiment provides a similar image retrieval device 400, including: The acquisition module 401 is used to acquire the image to be retrieved and the target image, and input the image to be retrieved and the target image into a pre-trained image segmentation model to acquire a first segmentation result corresponding to the image to be retrieved and a second segmentation result corresponding to the target image; the first segmentation result includes the panorama segmentation result to be retrieved and the high-level semantic features to be retrieved, and the second segmentation result includes the panorama segmentation result of the target and the high-level semantic features of the target. The retrieval calculation module 402 is used to perform intra-entity weighted feature calculation on the first segmentation result to obtain the entity description feature set to be retrieved, and to perform geometric pair spatial topological feature calculation on the first segmentation result to obtain the spatial topological feature set to be retrieved. The target calculation module 403 is used to perform intra-entity weighted feature calculation on the second segmentation result to obtain a target entity description feature set, and to perform geometric pair space topological feature calculation on the second segmentation result to obtain a target space topological feature set; The entity description calculation module 404 is used to calculate the similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set, so as to obtain the entity description similarity. The spatial topology calculation module 405 is used to calculate the similarity between the spatial topology features of geometric pairs with the same topological relationship type in the spatial topology feature set to be retrieved and the target spatial topology feature set, and obtain the spatial topology similarity. The weighted fusion module 406 is used to perform weighted fusion of the entity description similarity and the spatial topology similarity to obtain an overall similarity, and compare the overall similarity with a predetermined matching threshold; The determination module 407 is used to determine that the image to be retrieved and the target image are successfully matched when the overall similarity is greater than the predetermined matching threshold.

[0079] Furthermore, the search calculation module 402 includes: The entity extraction unit is used to extract all entities to be retrieved from the panoramic segmentation results to be retrieved; The feature vector acquisition unit is used to acquire the feature vector corresponding to each pixel in the area covered by the target entity from the target high-level semantic features for each target entity, and to form a target point set by combining the feature vectors of all pixels in the area covered by the target entity. The coordinate calculation unit is used to calculate the position coordinates of each pixel in the set of points to be retrieved and to determine the geometric center point of the entity to be retrieved. The allocation unit is used to assign a weight to each pixel in the set of points to be retrieved, with the geometric center point as a reference. The weighted average unit is used to perform a weighted average of the feature vectors of all pixels in the set of points to be retrieved according to their respective assigned weights, so as to obtain the entity description features of the entity to be retrieved. The feature aggregation unit is used to aggregate the entity description features of all entities to be retrieved to obtain the entity description feature set to be retrieved. The first entity pair extraction unit is used to extract all entity pairs from the panorama segmentation result to be retrieved, and each entity pair contains two different entities to be retrieved. The model confirmation unit is used to confirm the overlapping areas between the interior, boundary and exterior of the two entities to be retrieved in the entity pair through the spatial topology relationship model for each entity pair. The average calculation unit is used to obtain the feature vectors corresponding to all pixels in each overlapping region from the high-level semantic features to be retrieved, and to perform a weighted average calculation on all feature vectors in each overlapping region to obtain the region feature vector corresponding to each overlapping region. An arrangement unit is used to arrange all the regional feature vectors according to a predetermined row and column order to obtain the spatial topological features of the entity pair; The feature combination unit is used to combine the spatial topological features of all entity pairs to obtain the spatial topological feature set to be retrieved.

[0080] Furthermore, the target calculation module 403 includes: The target entity extraction unit is used to extract all target entities from the target panoramic segmentation result; The target point set construction unit is used to obtain the feature vector corresponding to each pixel in the area covered by the target entity from the target high-level semantic features for each target entity, and to construct a target point set by combining the feature vectors of all pixels in the area covered by the target entity. The center point determination unit is used to calculate the position coordinates of each pixel in the target point set and determine the geometric center point of the target entity; A weight allocation unit is used to assign a weight to each pixel in the target point set, with the geometric center point as a reference. The entity description feature acquisition unit is used to perform a weighted average of the feature vectors of all pixels in the target point set according to their respective assigned weights to obtain the entity description features of the target entity. The target entity description feature set acquisition unit is used to summarize the entity description features of all target entities to obtain the target entity description feature set; The second entity pair extraction unit is used to extract all entity pairs from the target panoramic segmentation result, and each entity pair contains two different target entities; The region confirmation unit is used to confirm the overlapping areas between the interior, boundary and exterior of the two target entities of the entity pair for each entity pair through a spatial topology relationship model. The region feature vector acquisition unit is used to acquire the feature vectors corresponding to all pixels in each overlapping region from the target high-level semantic features, and to perform a weighted average calculation on all feature vectors in each overlapping region to obtain the region feature vector corresponding to each overlapping region. The sequential arrangement unit is used to arrange all the regional feature vectors according to a predetermined row and column order to obtain the spatial topological features of the entity pair; The target space topology feature set acquisition unit is used to combine the spatial topology features of all entity pairs to obtain the target space topology feature set.

[0081] Furthermore, the entity description calculation module 404 includes: The distance calculation unit is used to calculate the cosine distance between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category entities in the target entity description feature set; A distance determination unit is used to determine whether the cosine distance is less than a first predetermined distance threshold. The failure determination unit is used to determine that the matching has failed if the cosine distance is greater than or equal to the first predetermined distance threshold. The success determination unit is used to determine that the match is successful if the cosine distance is less than the first predetermined distance threshold. The mean calculation unit is used to summarize the cosine distances of all successfully matched entities and calculate the mean of the cosine distances of all successfully matched entities to obtain the entity description similarity.

[0082] Furthermore, the mean calculation unit includes: The coordinate extraction subunit is used to extract the geometric center coordinates of the entities corresponding to each pair of successfully matched entity description features in their respective images, forming a set of matching point pairs. The fitting subunit is used to randomly iteratively select a pair of matching points to fit a spatial transformation matrix; An error calculation subunit is used to calculate the projection error of the remaining matching point pairs under the spatial transformation matrix. A marking subunit is used to mark matching point pairs whose projection error is less than or equal to a predetermined error threshold as interior points, and matching point pairs whose projection error is greater than the predetermined error threshold as exterior points; The aggregation sub-unit is used to aggregate all interior points in this iteration and calculate their corresponding number to obtain the set of interior points and the number of interior points. The reserved sub-unit is used to retain the set of interior points corresponding to the iteration with the most interior points when a preset number of iterations is reached. Each interior point in the set of interior points corresponds to a pair of successfully matched entity description features.

[0083] Furthermore, the mean calculation unit also includes: The statistics subunit is used to count the number of successful matches and obtain the match count. The first determination subunit is used to determine that the image to be retrieved and the target image fail to match if the number of matches is 0. The similarity determination subunit is used to determine whether the entity description similarity is greater than or equal to a second predetermined distance threshold if the number of matches is 1. The second determination subunit is used to determine that the image to be retrieved and the target image fail to match if the similarity of the entity description is greater than or equal to a second predetermined distance threshold. The third determination subunit is used to determine that the image to be retrieved and the target image are successfully matched if the similarity of the entity description is less than a second predetermined distance threshold. The similarity calculation subunit is used to calculate the similarity between the geometric pairs of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set if the number of matches is greater than 1, so as to obtain the spatial topological similarity.

[0084] Furthermore, the spatial topology calculation module 405 includes: The type extraction unit is used to extract each geometric pair spatial topology feature and its corresponding first entity category label, second entity category label and topology relationship type from the set of spatial topology features to be retrieved; The label extraction unit is used to extract each geometric pair spatial topological feature and its corresponding first entity category label, second entity category label and topological relationship type from the target spatial topological feature set; The tag filtering unit is used to filter out geometric pairs of spatial topology features that are the same as the first entity category label, the second entity category label, and the topological relationship type in the spatial topology feature set to be retrieved and the target spatial topology feature set. The cosine distance calculation unit is used to calculate the cosine distance between the region feature vectors corresponding to the two geometric pairs of spatial topological features in each geometric pair in row and column order, and obtain multiple similarity values. The accumulation unit is used to accumulate all similarity values ​​calculated for all geometric pairs of spatial topological feature pairs and then divide by the total number of similarity values ​​to obtain the spatial topological similarity.

[0085] This invention provides a similar image retrieval device. First, it acquires a target image and a image to be retrieved. Then, it inputs the target image and the image to be retrieved into a pre-trained image segmentation model to obtain a first segmentation result for the target image and a second segmentation result for the image to be retrieved. The first segmentation result includes a panoramic segmentation result and high-level semantic features to be retrieved, while the second segmentation result includes a panoramic segmentation result and high-level semantic features of the target image. Intra-entity weighted feature calculation is performed on the first segmentation result to obtain a target entity description feature set, and geometric pair spatial topological feature calculation is performed on the first segmentation result to obtain a target spatial topological feature set. Intra-entity weighted feature calculation is performed on the second segmentation result to obtain a target entity description feature set. The second segmentation result is used to calculate the spatial topological features of geometric pairs to obtain the target spatial topological feature set. The similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set is calculated to obtain the entity description similarity. The similarity between geometric pairs of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity. The entity description similarity and spatial topological similarity are then weighted and fused to obtain the overall similarity, which is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, the image to be retrieved and the target image are considered to have matched successfully. Semantic entities in the image are obtained using panoramic segmentation, and compact features for each entity are generated through weighted averaging. Entity-level features replace massive amounts of local keypoint features, significantly reducing storage requirements while preserving rich detail information. Furthermore, the spatial topological features between entity pairs are calculated and matched using geometric pairs of spatial topological features, enhancing the ability to judge the consistency of the internal structural layout of the image. Meanwhile, by combining entity feature similarity and spatial topological similarity for retrieval, entity features are more semantically distinguishable, and combined with spatial constraints, the retrieval accuracy and robustness in complex scenarios are significantly improved, achieving a superior balance between storage efficiency and representation capability.

[0086] Specific limitations regarding the similar image retrieval device can be found in the limitations of the similar image retrieval method described above, and will not be repeated here. Each unit in the aforementioned similar image retrieval device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0087] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a similar image retrieval method on the server side.

[0088] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of a similar image retrieval method.

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The system acquires a target image and a search image, and inputs the target image and the search image into a pre-trained image segmentation model to obtain a first segmentation result corresponding to the search image and a second segmentation result corresponding to the target image. The first segmentation result includes a search panorama segmentation result and a search high-level semantic feature, and the second segmentation result includes a target panorama segmentation result and a target high-level semantic feature. The first segmentation result is subjected to intra-entity weighted feature calculation to obtain the entity description feature set to be retrieved, and the first segmentation result is subjected to geometric pair spatial topological feature calculation to obtain the spatial topological feature set to be retrieved. The target entity description feature set is obtained by performing intra-entity weighted feature calculation on the second segmentation result, and the target spatial topological feature set is obtained by performing geometric pair spatial topological feature calculation on the second segmentation result. The similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set is calculated to obtain the entity description similarity. The similarity of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity. The entity description similarity and the spatial topology similarity are weighted and fused to obtain the overall similarity, and the overall similarity is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, it is determined that the image to be retrieved and the target image are successfully matched.

[0090] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The system acquires a target image and a search image, and inputs the target image and the search image into a pre-trained image segmentation model to obtain a first segmentation result corresponding to the search image and a second segmentation result corresponding to the target image. The first segmentation result includes a search panorama segmentation result and a search high-level semantic feature, and the second segmentation result includes a target panorama segmentation result and a target high-level semantic feature. The first segmentation result is subjected to intra-entity weighted feature calculation to obtain the entity description feature set to be retrieved, and the first segmentation result is subjected to geometric pair spatial topological feature calculation to obtain the spatial topological feature set to be retrieved. The target entity description feature set is obtained by performing intra-entity weighted feature calculation on the second segmentation result, and the target spatial topological feature set is obtained by performing geometric pair spatial topological feature calculation on the second segmentation result. The similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set is calculated to obtain the entity description similarity. The similarity of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity. The entity description similarity and the spatial topology similarity are weighted and fused to obtain the overall similarity, and the overall similarity is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, it is determined that the image to be retrieved and the target image are successfully matched.

[0091] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, database, or their media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0094] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. The user personal information involved in the embodiments of this application is obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0095] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A similar image retrieval method, characterized in that, include: Acquire a search image and a target image, and input the search image and the target image into a pre-trained image segmentation model to obtain a first segmentation result corresponding to the search image and a second segmentation result corresponding to the target image; The first segmentation result includes the panorama segmentation result to be retrieved and the high-level semantic features to be retrieved; the second segmentation result includes the panorama segmentation result of the target and the high-level semantic features of the target. The first segmentation result is subjected to intra-entity weighted feature calculation to obtain the entity description feature set to be retrieved, and the first segmentation result is subjected to geometric pair spatial topological feature calculation to obtain the spatial topological feature set to be retrieved. The target entity description feature set is obtained by performing intra-entity weighted feature calculation on the second segmentation result, and the target spatial topological feature set is obtained by performing geometric pair spatial topological feature calculation on the second segmentation result. The similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set is calculated to obtain the entity description similarity. The similarity of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity. The entity description similarity and the spatial topology similarity are weighted and fused to obtain the overall similarity, and the overall similarity is compared with a predetermined matching threshold. When the overall similarity is greater than the predetermined matching threshold, it is determined that the image to be retrieved and the target image are successfully matched.

2. The similar image retrieval method according to claim 1, characterized in that, The step of performing intra-entity weighted feature calculation on the first segmentation result to obtain the entity description feature set to be retrieved, and performing geometric pair spatial topological feature calculation on the first segmentation result to obtain the spatial topological feature set to be retrieved includes: Extract all entities to be retrieved from the panoramic segmentation results to be retrieved; For each entity to be retrieved, the feature vector corresponding to each pixel in the area covered by the entity to be retrieved is obtained from the high-level semantic features to be retrieved, and the feature vectors of all pixels in the area covered by the entity to be retrieved are used to form a set of points to be retrieved; Calculate the position coordinates of each pixel in the set of points to be retrieved and determine the geometric center point of the entity to be retrieved; Based on the geometric center point, assign a weight to each pixel in the set of points to be retrieved; The feature vectors of all pixels in the set of points to be retrieved are weighted and averaged according to their respective assigned weights to obtain the entity description features of the entity to be retrieved. The entity description features of all entities to be retrieved are summarized to obtain the entity description feature set to be retrieved; Extract all entity pairs from the panorama segmentation results to be retrieved, each entity pair containing two different entities to be retrieved; For each entity pair, the overlapping areas between the interior, boundary and exterior of the two entities to be retrieved in the entity pair are identified through a spatial topology relationship model; The feature vectors corresponding to all pixels in each overlapping region are obtained from the high-level semantic features to be retrieved, and a weighted average calculation is performed on all feature vectors in each overlapping region to obtain the region feature vector corresponding to each overlapping region. Arrange all the region feature vectors according to a predetermined row and column order to obtain the spatial topological features of the entity pair; The spatial topological features of all entity pairs are combined to obtain the set of spatial topological features to be retrieved.

3. The similar image retrieval method according to claim 1, characterized in that, The step of calculating the target entity description feature set by performing intra-entity weighted feature calculation on the second segmentation result and calculating the target spatial topological feature set by performing geometric pair spatial topological feature calculation on the second segmentation result includes: Extract all target entities from the aforementioned panoramic segmentation results; For each target entity, the feature vector corresponding to each pixel in the area covered by the target entity is obtained from the high-level semantic features of the target, and the feature vectors of all pixels in the area covered by the target entity are used to form a target point set; Calculate the position coordinates of each pixel in the target point set and determine the geometric center point of the target entity; Based on the geometric center point, assign a weight to each pixel in the target point set; The feature vectors of all pixels in the target point set are weighted and averaged according to their respective assigned weights to obtain the entity description features of the target entity. The entity description features of all target entities are summarized to obtain the target entity description feature set; Extract all entity pairs from the target panoramic segmentation results, where each entity pair contains two different target entities; For each entity pair, the overlapping areas between the interiors, boundaries, and exteriors of the two target entities in the entity pair are identified using a spatial topology relationship model. The feature vectors corresponding to all pixels in each overlapping region are obtained from the target high-level semantic features, and a weighted average calculation is performed on all feature vectors in each overlapping region to obtain the region feature vector corresponding to each overlapping region. Arrange all the region feature vectors according to a predetermined row and column order to obtain the spatial topological features of the entity pair; The spatial topological features of all entity pairs are combined to obtain the target spatial topological feature set.

4. The similar image retrieval method according to claim 1, characterized in that, The step of calculating the similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set to obtain the entity description similarity includes: Calculate the cosine distance between each entity description feature in the entity description feature set to be retrieved and the entity description features of the same category in the target entity description feature set; Determine whether the cosine distance is less than a first predetermined distance threshold; If the cosine distance is greater than or equal to the first predetermined distance threshold, the matching is determined to have failed; If the cosine distance is less than the first predetermined distance threshold, then the match is considered successful; Sum the cosine distances of all successful matches and calculate the mean of all successful cosine distances to obtain the entity description similarity.

5. The similar image retrieval method according to claim 4, characterized in that, The process of summarizing all successfully matched cosine distances and calculating the mean of all successfully matched cosine distances to obtain the entity description similarity includes: Extract the geometric center coordinates of the entities corresponding to each pair of successfully matched entity description features in their respective images to form a set of matching point pairs; A spatial transformation matrix is ​​fitted by randomly iterating to select a pair of matching points; Calculate the projection error of the remaining matching point pairs under the spatial transformation matrix; Matching point pairs whose projection error is less than or equal to a predetermined error threshold are marked as interior points, and matching point pairs whose projection error is greater than the predetermined error threshold are marked as exterior points. Gather all interior points from this iteration and calculate their corresponding counts to obtain the interior point set and the number of interior points; When the preset number of iterations is reached, the set of inliers corresponding to the iteration with the most inliers is retained. Each inlier in the set of inliers corresponds to a pair of successfully matched entity description features.

6. The similar image retrieval method according to claim 4, characterized in that, The process of summarizing all successfully matched cosine distances, calculating the mean of all successfully matched cosine distances, and obtaining the entity description similarity includes: Count the number of successful matches to obtain the total number of matches; If the number of matches is 0, then the image to be retrieved and the target image are determined to be a failed match. If the number of matches is 1, then determine whether the entity description similarity is greater than or equal to the second predetermined distance threshold; If the entity description similarity is greater than or equal to the second predetermined distance threshold, then the image to be retrieved and the target image are determined to be mismatched. If the entity description similarity is less than a second predetermined distance threshold, then the image to be retrieved and the target image are determined to be a successful match. If the number of matches is greater than 1, then the similarity of the spatial topological features of geometric pairs with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set is calculated to obtain the spatial topological similarity.

7. The similar image retrieval method according to claim 1, characterized in that, The step of calculating the similarity between geometric pairs of spatial topological features with the same topological relationship type in the spatial topological feature set to be retrieved and the target spatial topological feature set to obtain spatial topological similarity includes: Extract each geometric pair spatial topological feature and its corresponding first entity category label, second entity category label, and topological relationship type from the set of spatial topological features to be retrieved; Extract each geometric pair spatial topological feature and its corresponding first entity category label, second entity category label, and topological relationship type from the target spatial topological feature set; Filter out geometric pairs of spatial topological features in the spatial topological feature set to be retrieved that have the same first entity category label, the same second entity category label, and the same topological relationship type as those in the target spatial topological feature set. For each geometric pair of spatial topological features, the cosine distance between the region feature vectors corresponding to the two geometric pairs of spatial topological features is calculated in row and column order to obtain multiple similarity values. The spatial topological similarity is obtained by summing all the similarity values ​​calculated for all geometric pairs of spatial topological features and dividing by the total number of similarity values.

8. A similar image retrieval device, characterized in that, include: The acquisition module is used to acquire a search image and a target image, and input the search image and the target image into a pre-trained image segmentation model to acquire a first segmentation result corresponding to the search image and a second segmentation result corresponding to the target image; the first segmentation result includes a search panorama segmentation result and a search high-level semantic feature, and the second segmentation result includes a target panorama segmentation result and a target high-level semantic feature; The retrieval calculation module is used to perform intra-entity weighted feature calculation on the first segmentation result to obtain the entity description feature set to be retrieved, and to perform geometric pair spatial topological feature calculation on the first segmentation result to obtain the spatial topological feature set to be retrieved. The target calculation module is used to perform intra-entity weighted feature calculation on the second segmentation result to obtain a target entity description feature set, and to perform geometric pair space topological feature calculation on the second segmentation result to obtain a target space topological feature set; The entity description calculation module is used to calculate the similarity between each entity description feature in the entity description feature set to be retrieved and the entity description features of entities of the same category in the target entity description feature set, and obtain the entity description similarity. The spatial topology calculation module is used to calculate the similarity between the spatial topology features of geometric pairs with the same topological relationship type in the spatial topology feature set to be retrieved and the target spatial topology feature set, and to obtain the spatial topology similarity. The weighted fusion module is used to perform weighted fusion of the entity description similarity and the spatial topology similarity to obtain an overall similarity, and compare the overall similarity with a predetermined matching threshold; The determination module is used to determine that the image to be retrieved and the target image are successfully matched when the overall similarity is greater than the predetermined matching threshold.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the similar image retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the similar image retrieval method as described in any one of claims 1 to 7.

Images