Image-text cross-modal retrieval method and system based on high-dimensional ball embedding

Abstract

The invention discloses an image-text cross-modal retrieval method and system based on high-dimensional ball embedding, and the method comprises the steps: carrying out the feature extraction processing of a target image and a target text through a backbone network and a word embedding method based on the target image and the target text, and obtaining a visual feature and a text feature; based on a ball encoder, performing center calculation and semantic uncertainty modeling of ball embedding to obtain a visual ball center vector, a visual uncertainty radius, a text ball center vector and a text uncertainty radius; similarity learning is carried out on the center vector and radius of the ball through a Monte Carlo sampling method of ball embedding, and image text cross-modal retrieval is achieved. According to the method, cross-modal alignment is enhanced through semantic uncertainty and diversity between visual texts, so that the cross-modal retrieval precision of the image texts is improved. The image-text cross-modal retrieval method and system based on high-dimensional ball embedding can be widely applied to the technical field of image-text retrieval.

Description

Technical Field

[0001] This invention relates to the field of image and text retrieval technology, and in particular to an image-text cross-modal retrieval method and system based on high-dimensional sphere embedding. Background Technology

[0002] In the field of intelligent supervision, image-text cross-modal retrieval is a core task, aiming to measure the semantic relevance between images and text descriptions. It is widely used in various supervision processes, such as retrieving and comparing construction site images with regulations and industry standards in the field of building engineering, and retrieving historical supervision images based on text descriptions. Besides image-text retrieval for intelligent supervision, this task can also support various application scenarios such as visual language navigation and visual question answering. However, bridging the gap between visual and text modalities remains inherently challenging: images typically contain rich but ambiguous signals (such as cluttered backgrounds, object occlusion, and diverse object configurations), while text descriptions are often concise, discrete, and semantically selective. This modal difference is particularly prominent in one-to-many matching scenarios: a single image may correspond to multiple valid text interpretations, varying in specificity and abstraction; and vice versa.

[0003] Despite significant progress in image-text retrieval, a key challenge remains: how to construct a unified embedding space to effectively bridge the semantic gap between low-level entangled visual content and high-level sparse textual semantics, especially in a one-to-many correspondence setting. This difficulty stems from the inherent asymmetry of multimodal data: visual signals are continuous, spatially dense, and viewpoint-dependent; while language is symbolic, composable, and inherently discrete. Furthermore, visual data encapsulates rich, context-dependent information (such as texture, shape, and spatial relationships), which may not directly correspond to the more abstract and categorical nature of textual representations. Therefore, without fine-grained modeling of these differences, the embedding space may fail to properly align the rich diversity of information in images with the often sparse and abstract content in text, resulting in incomplete or inaccurate semantic mapping.

[0004] In practice, one-to-many correspondences reflect the semantic uncertainty between images and text, stemming from inherent differences in human understanding and interpretation. Different individuals may perceive the same image in multiple ways; similarly, the same text may convey multiple meanings depending on the context. This variability is further amplified by the subjectivity of language and the contextual dependence of visual content. Therefore, modeling this semantic uncertainty is crucial, as it enables the system to consider multiple possibilities for image-text alignment. More importantly, explicitly incorporating such uncertainty into representation learning not only alleviates the limitations of deterministic embeddings but also allows retrieval models to better capture fine-grained semantics.

[0005] Related techniques have explored probabilistic and geometry-based embedding methods. For example, Gaussian probabilistic embedding models modal ambiguity by learning the mean and variance, but it still fundamentally relies on the estimation and alignment of cross-modal directional cues. This alignment is highly sensitive to rotation, object layout, and viewpoint changes, thus limiting its performance under real-world visual variations. Furthermore, geometric representations such as boxes and sectors attempt to provide modality-oriented uncertainty modeling. However, these methods still fundamentally depend on matching directional information between heterogeneous modalities. When visual inputs exhibit rotational variations, occlusion, or incomplete label matching, this dependence can easily lead to misalignment in the embedding space, thereby degrading retrieval performance. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide an image-text cross-modal retrieval method and system based on high-dimensional sphere embedding. This method enhances cross-modal alignment by leveraging the semantic uncertainty and diversity between visual texts, thereby improving the accuracy of image-text cross-modal retrieval.

[0007] The first technical solution adopted in this invention is: an image-text cross-modal retrieval method based on high-dimensional sphere embedding, comprising the following steps: Based on the target image and target text, feature extraction is performed on the target image and target text respectively through backbone network and word embedding methods to obtain visual features and text features; Based on the ball encoder, the center calculation of sphere embedding and semantic uncertainty modeling are performed on visual features and text features respectively, to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius; By employing a Monte Carlo sampling method based on sphere embedding to learn the similarity between the visual sphere center vector, the visual uncertainty radius, the text sphere center vector, and the text uncertainty radius, cross-modal image-text retrieval is achieved.

[0008] Furthermore, the step of calculating the center of sphere embedding and modeling semantic uncertainty for visual and textual features based on the sphere encoder, respectively, to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector, and text uncertainty radius, specifically includes: Based on the visual sphere encoder, the visual features are used to perform center calculation of sphere embedding and semantic uncertainty modeling through an attention-guided entropy mechanism, resulting in the visual sphere center vector and visual uncertainty radius. Based on the text ball encoder, the text features are embedded into a ball and the semantic uncertainty is modeled by an attention-guided entropy mechanism to obtain the text ball center vector and the text uncertainty radius.

[0009] Furthermore, the step of calculating the center of the visual sphere embedding and modeling semantic uncertainty through an attention-guided entropy mechanism based on the visual sphere encoder to obtain the visual sphere center vector and the visual uncertainty radius specifically includes: The visual features are sequentially subjected to fully connected transformation and global average pooling extraction to obtain the global semantic information of the visual features and construct the first component. Dimension transformation is performed on deep features in visual features to obtain the dimensionally transformed deep features; After the dimensionality transformation, the deep features are activated by hyperbolic tangent and then the attention vectors are generated through a fully connected layer. The attention weights of each visual category feature are obtained by normalization through the Softmax function. The deep features after dimensionality transformation are multiplied by the attention weights and then activated by the Sigmoid activation function to generate the second component; By combining the first and second components, a visual sphere center vector is constructed; The semantic uncertainty radius is obtained by measuring the information entropy of the attention weight distribution.

[0010] Furthermore, the expression for calculating the center of the sphere embedding of visual features using the attention-guided entropy mechanism is as follows: ; ; ; ; In the above formula, Indicates the first component. Indicates the second component, Indicates a fully connected transformation. Indicates global average pooling. Indicates visual characteristics, Represents the attention vector. This represents the Softmax function. This represents the hyperbolic tangent activation function. Represents the dimensional transformation function. This represents the center vector of the visual sphere.

[0011] Furthermore, the expression for measuring semantic uncertainty based on the information entropy of the attention weight distribution is as follows: ; In the above formula, Represents the radius of visual uncertainty. This represents a hyperparameter that modulates the intensity of visual semantic uncertainty. This represents the attention weights in the attention vector corresponding to the visual features. Indicates the radius.

[0012] Furthermore, the step of calculating the center of the text sphere embedding and modeling semantic uncertainty through an attention-guided entropy mechanism based on the text sphere encoder to obtain the text sphere center vector and the text uncertainty radius specifically includes: Contextual modeling of text features is performed using a bidirectional GRU network to capture bidirectional semantic information. Attention features are extracted from text features using two fully connected layers and the activation function tanh, generating word-level attention distribution weights. The text features are multiplied by the word-level attention distribution weights, and then weighted features are generated using the Sigmoid activation function. By combining bidirectional semantic information and weighted features, a text sphere center vector is constructed; The semantic uncertainty radius is obtained by measuring the information entropy of the distribution of word-level attention weights.

[0013] Furthermore, the step of performing similarity learning on the visual sphere center vector, visual uncertainty radius, text sphere center vector, and text uncertainty radius using the Monte Carlo sampling method with sphere embedding to achieve cross-modal image-text retrieval specifically includes: Determine the center coordinates of the spheres based on the visual sphere center vector and the text sphere center vector, determine the radius based on the visual uncertainty radius and the text uncertainty radius, set the batch size, embedding dimension and sampling number, initialize an empty list and set the loop counter to 1; Determine if the loop counter is less than or equal to the batch size. If so, extract the center coordinates of the i-th sample from the sphere center coordinates and extract the radius value corresponding to the i-th sample from the radius. Sample N d-dimensional direction vectors from the standard normal distribution, construct a direction matrix, and calculate the L2 norm of each row vector to obtain the norm vector; Divide the norm vector of each row of the direction matrix by its corresponding L2 norm to obtain the normalized direction matrix; Sample N scalar values ​​from a uniform distribution to obtain a uniform sampling vector, and multiply each element of the uniform sampling vector by the corresponding radius value to obtain a scaled radius vector; The scaled radius vector is multiplied element-wise with the normalized direction matrix to obtain the offset vector relative to the center of the sphere. The offset vector is added element by element to the center coordinates of the sphere to obtain the final set of sampling points. The set of sampling points is then added to a list, and the loop counter is incremented by 1 until the loop counter is greater than the batch size. All sampling points in the list are stacked along the batch dimension to construct the final sampling point tensor, enabling cross-modal retrieval of image text.

[0014] Furthermore, the loss function of the sphere embedding Monte Carlo sampling method includes a matching loss and a maximum penetration depth loss, wherein the expression for the matching loss is as follows: ; ; In the above formula, This represents the matching loss function. This represents the maximum sigmoid similarity among all sampled pairs. , Representing the sampling in the image sphere and text sphere respectively. and One point, This represents the sigmoid function. , Indicates learnable parameters, Represents the L2 norm. This represents the number of samples for each ball. This indicates the magnitude of the text feature matching in the sampled image. , Features indicating the use of image text This represents conditional probability.

[0015] Furthermore, the expression for the maximum penetration depth loss is as follows: ; ; In the above formula, This represents the maximum penetration depth loss function. State of expression The ball One sampling point, , Representing modes The center and radius of the corresponding instance sphere, Indicates measurement from modality To mode Maximum penetration depth Indicates hyperparameters, This represents the number of samples for each ball. This represents the penetration depth from mode A to mode B. Indicates the first Each sampling feature.

[0016] The second technical solution adopted in this invention is: an image-text cross-modal retrieval system based on high-dimensional sphere embedding, comprising: The first module is used to extract features from the target image and target text respectively using a backbone network and word embedding methods to obtain visual features and text features. The second module is used to perform center calculation and semantic uncertainty modeling of sphere embedding on visual features and text features respectively, based on the sphere encoder, to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius; The third module is used to perform similarity learning on the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius through the Monte Carlo sampling method of sphere embedding, so as to realize cross-modal retrieval of images and text.

[0017] The beneficial effects of the method and system of this invention are as follows: Based on the target image and target text, this invention performs feature extraction processing on the target image and target text respectively through backbone network and word embedding method to obtain visual features and text features; further, based on sphere encoder, it performs center calculation of sphere embedding and semantic uncertainty modeling on visual features and text features respectively to obtain visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius. Through a high-dimensional sphere embedding framework, semantic uncertainty is modeled in a unified and isotropic embedding space. By representing cross-modal instances as high-dimensional spheres instead of points, the inherent diversity and ambiguity in one-to-many image-text matching scenarios are naturally captured, thereby improving robustness and retrieval accuracy. Finally, a Monte Carlo sampling method with sphere embedding is used to learn the similarity of the visual sphere center vector, visual uncertainty radius, text sphere center vector, and text uncertainty radius to achieve cross-modal image-text retrieval. The center and radius of the semantic sphere are learned through an attention-guided entropy mechanism. Furthermore, a Monte Carlo sampling strategy is proposed, and a scalable semantic relevance loss based on maximum penetration depth is introduced to achieve more discriminative cross-modal matching and improve the accuracy of cross-modal image-text retrieval. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of an image-text cross-modal retrieval method based on high-dimensional sphere embedding according to the present invention. Figure 2 This is a structural block diagram of an image-text cross-modal retrieval system based on high-dimensional sphere embedding according to the present invention; Figure 3 This is a schematic diagram of a high-dimensional sphere embedding framework for multimodal information modeling provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of a high-dimensional sphere embedding framework provided in a specific embodiment of the present invention; Figure 5 This is a schematic diagram comparing semantic uncertainty and RP provided in a specific embodiment of the present invention; Figure 6 This is a schematic diagram comparing semantic uncertainty with R@K provided in a specific embodiment of the present invention; Figure 7 This is a schematic diagram comparing the maximum penetration depth loss with the RP provided in a specific embodiment of the present invention; Figure 8 This is a schematic diagram comparing the loss of maximum penetration depth with R@K provided in a specific embodiment of the present invention; Figure 9 This is a schematic diagram of RP comparison for N pairs of cross-modal retrievals provided in a specific embodiment of the present invention; Figure 10 This is a schematic diagram of R@K comparison for N pairs of cross-modal retrieval provided in a specific embodiment of the present invention; Figure 11 This is a schematic diagram of the T-SNE visualization results of high-dimensional sphere embedding provided in a specific embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0020] First, it's important to note that image-text cross-modal retrieval is a core task in the field of intelligent supervision, with its key technology lying in mapping images and text into a unified embedding space. Most current image-text retrieval methods employ deterministic single-point embedding, focusing only on labeled positive sample pairs (Ground-truth pairs) while neglecting "one-to-many" scenarios—where a query often corresponds to multiple semantically similar instances. Although recent geometrically based embedding methods have provided directional solutions for capturing the diversity of matches, they inherently rely on cross-modal orientation information estimation and alignment, making them sensitive to rotation and viewpoint changes. Furthermore, these methods still have limitations in fully utilizing "one-to-many" correspondences.

[0021] Furthermore, this paper describes the relevant technologies at this stage. Image-text retrieval aims to bridge the semantic gap between visual and textual modalities by identifying semantically aligned image-text pairs from heterogeneous data sources. Based on different feature representation strategies, existing methods are generally divided into two main categories: global-based methods and region-based methods. Global methods typically map the entire image and the entire text into a unified embedding space and use a distance metric to measure their semantic relevance. For example, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are used to encode images and text respectively; graph convolutional networks (GCNs) are employed to enhance visual and textual feature representations, and inner products are used to calculate the relevance between the two modalities. In contrast, region-based methods first use object detection tools to identify objects in the image, and then use attention mechanisms to establish fine-grained semantic correspondences between visual regions and text elements. Stacked Attention Networks are representative works in this direction. Building upon this, various attention-based architectures have been proposed to better capture the fine-grained alignment between images and text. In general, global methods are computationally efficient but often overlook fine-grained semantic details; while region methods capture local visual-text alignments through attention mechanisms, achieving more precise retrieval. However, despite their significant success in retrieving the most relevant instances, these methods still have significant limitations when handling one-to-many scenarios. This deficiency mainly stems from their reliance on rigid point-to-point alignment strategies—strategies that tend to focus only on exact matches while ignoring multiple semantically plausible and relevant candidate results. Therefore, designing a retrieval framework that can explicitly accommodate the inherent one-to-many relationship between visual and textual data is crucial.

[0022] One-to-many matching poses a significant challenge in image-text retrieval because a single query may correspond to multiple semantically diverse and context-dependent candidate results. Theoretically, one-to-many associations more closely resemble how humans understand images and text. For the same image, different observers may focus on different aspects, resulting in different textual descriptions; similarly, the inherent ambiguity and generalization of language often lead to a single text corresponding to multiple images. Early methods such as PVSE (Polysemous Visual-Semantic Embedding) generated multiple polysemous embeddings for each instance through multi-head self-attention mechanisms and global-local feature fusion; while PCME (Probabilistic Cross-Modal Embedding) modeled each instance as a Gaussian distribution and estimated the matching probability through sampling, thus providing an interpretable representation of uncertainty. Subsequently, geometric embedding methods attempted to encode features into bounded regions to explicitly express semantic diversity. For example, P2RM (Point-to-Rectangle Matching) introduces axis-aligned rectangle embeddings and point-to-rectangle distance metrics, handling one-to-many relationships without dense sampling. This method was subsequently extended by Geometric Matching, which achieves rectangle-to-rectangle alignment by calculating the intersection volume between rectangles. More recently, ESSE (Estimating Semantics via SectorEmbedding) employs sector region embeddings (defined by the center and the subtended angle), providing a bounded and rotation-sensitive representation that, compared to Gaussian or rectangular methods, better encodes both semantic mean and uncertainty simultaneously.

[0023] Unlike the methods described above, the HDBE framework proposed in this embodiment encapsulates image and text representations within a high-dimensional sphere. Theoretically, representing multimodal instances as a high-dimensional sphere offers several advantages. First, it provides an isotropic and rotation-invariant structure, ensuring consistent semantic similarity regardless of orientation in the latent space. Second, this design naturally accommodates the inherent variability in human understanding, as different observers may emphasize different aspects of the same image or produce different textual descriptions. By modeling this variability of the sphere's radius and exploring it through sampling, HDBE effectively captures the semantic diversity behind one-to-many correlations.

[0024] Based on this, such as Figure 3As shown, this embodiment leverages the rotational invariance and isotropic properties of high-dimensional spheres to propose High-Dimensional Ball Embedding (HDBE): representing each instance in different modalities as a high-dimensional sphere and effectively modeling a "one-to-many" correspondence by sampling points within the sphere. Specifically, this embodiment employs an end-to-end approach to simultaneously learn the center coordinates and radius of the high-dimensional sphere, where the radius reflects the semantic uncertainty of the image or text. Based on this, Monte Carlo sampling is used within the sphere to capture and represent the diversity of visual-textual semantics. Furthermore, a novel maximum penetration depth loss is introduced to enhance cross-modal alignment. Extensive experiments on the CUB and MSCOCO benchmark datasets demonstrate that this method exhibits superiority and effectiveness across multiple evaluation metrics.

[0025] Reference Figure 1 This invention provides an image-text cross-modal retrieval method based on high-dimensional sphere embedding, the method comprising the following steps: S100. Based on the target image and target text, feature extraction processing is performed on the target image and target text respectively through backbone network and word embedding method to obtain visual features and text features; Specifically, based on the target image and target text, feature extraction is performed on the target image and target text respectively through backbone network and word embedding methods to obtain visual features and text features.

[0026] In this embodiment, for visual representation, given an image Using ResNet to obtain visual features ,in This refers to the backbone network.

[0027] Similarly, given a text T containing L words, GloVe is used to extract text features. ,in This indicates the Glove embedding function.

[0028] S200, based on the ball encoder, performs center calculation of ball embedding and semantic uncertainty modeling on visual features and text features respectively, to obtain visual ball center vector, visual uncertainty radius, text ball center vector and text uncertainty radius; First, such as Figure 4As shown, to better capture the semantic uncertainty of multimodal data and reduce sensitivity to modeling modality-specific directional cues, we introduce a sphere encoder framework. In this framework, semantically reliable representations are first extracted from the visual and textual modalities using a Global Average Pooling (GAP) network and a Bidirectional GRU (Bi-GRU), respectively. These representations are then combined with deep features obtained from multi-layer networks to form the center of each high-dimensional sphere. Semantic uncertainty is modeled as a radius and adaptively determined through an attention-guided entropy mechanism. This design produces isotropic and interpretable representations, thereby promoting robust cross-modal alignment in retrieval tasks.

[0029] Specifically, given a visual feature map and text feature sequences Our ball encoder learns the center and radius for each modality to construct a high-dimensional ball embedding.

[0030] S210. Based on the visual sphere encoder, the visual features are processed by an attention-guided entropy mechanism to calculate the center of the sphere embedding and model the semantic uncertainty, thereby obtaining the visual sphere center vector and the visual uncertainty radius. Specifically, visual features are sequentially processed by fully connected transformation and global average pooling to extract global semantic information, thus constructing the first component. Deep features within the visual features undergo dimensionality transformation to obtain dimensionally transformed deep features. These dimensionally transformed deep features are then activated by hyperbolic tangent activation and passed through a fully connected layer to generate attention vectors. These vectors are then normalized using the Softmax function to obtain attention weights for each visual category feature. The dimensionally transformed deep features are multiplied by the attention weights and activated by the Sigmoid activation function to generate the second component. The first and second components are combined to construct the visual sphere center vector. Semantic uncertainty is measured based on the information entropy of the attention weight distribution to obtain the visual uncertainty radius.

[0031] In this embodiment, for the image modality, the center of the visual sphere is formed by the sum of two complementary components: the first component By performing a fully connected transformation on visual features followed by global average pooling, the most reliable global semantic information is captured; the second component Derived from deep feature representations, it retains richer structural and discriminative cues. By combining the two, the visual sphere center maintains stability while possessing strong expressive power, serving as a reliable anchor point for modeling semantic uncertainty. Therefore, the visual sphere center... This can be formally represented as: ; ; ; ; In the above formula, Indicates the first component. Indicates the second component, Indicates a fully connected transformation. Indicates global average pooling. Indicates visual characteristics, Represents the attention vector. This represents the Softmax function. This represents the hyperbolic tangent activation function. Represents the dimensional transformation function. This represents the center vector of the visual sphere.

[0032] in, This is the attention vector, which can be understood as the weights of features for each category; This is a dimensional transformation function; and These are the hyperbolic tangent and sigmoid activation functions, respectively.

[0033] In addition, radius This is achieved by first calculating the attention distribution and then measuring its information entropy. Intuitively, the sharper (concentrated) the attention distribution, the smaller the entropy, indicating more certain semantics; conversely, the flatter the distribution, the larger the entropy, reflecting higher uncertainty or ambiguity. This strategy provides an interpretable and adaptive way to model semantic uncertainty, allowing the radius to naturally encode the reliability of the visual representation. The entire process can be formalized as follows: ; In the above formula, Represents the radius of visual uncertainty. This represents a hyperparameter that modulates the intensity of visual semantic uncertainty. This represents the attention weights in the attention vector corresponding to the visual features. Indicates the radius.

[0034] in, Measuring visual semantic uncertainty; Hyperparameters used to adjust the intensity of visual semantic uncertainty; for The middle corresponds to Attention weights.

[0035] S220: Based on the text ball encoder, the text features are embedded into a ball and the semantic uncertainty is modeled by an attention-guided entropy mechanism to obtain the text ball center vector and the text uncertainty radius.

[0036] Specifically, a bidirectional GRU network is used to model the context of text features and capture bidirectional semantic information; attention features are extracted from text features through two fully connected layers and the tanh activation function to generate word-level attention distribution weights; text features are multiplied with word-level attention distribution weights and weighted features are generated through the Sigmoid activation function; the bidirectional semantic information and weighted features are combined to construct the text sphere center vector; and semantic uncertainty is measured based on the information entropy of the word-level attention distribution weights to obtain the text uncertainty radius.

[0037] In this embodiment, for the text modality, we adopt a design similar to that of the visual modality, the difference being that: to capture the semantic information of the text, we use a bidirectional GRU (Bi-GRU) to model context dependencies. Therefore, the center of the text sphere... It can be represented as: ; ; ; ; in This represents the word-level attention distribution. Radius Similarly, entropy is estimated using an attention-guided mechanism: ; in Reflecting the semantic uncertainty of the text, To adjust the hyperparameters for the intensity of semantic uncertainty in text, for The middle corresponds to Attention weights.

[0038] S300: By using the Monte Carlo sampling method of sphere embedding to perform similarity learning on the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius, cross-modal image-text retrieval is achieved.

[0039] First, it should be noted that to support cross-modal similarity learning for "one-to-many" image-text matching, we employ a Monte Carlo sampling strategy to generate sampling points within the high-dimensional spherical embedding. Specifically, given a batch of sphere centers... and the corresponding radius The sampling process is as follows: First, for each sample in the batch, N direction vectors are sampled from a standard multivariate normal distribution and normalized to unit vectors to obtain the directions within the sphere. Second, a scalar is sampled from a uniform distribution and multiplied by the radius of the corresponding sphere to obtain the sampling distance from the center of the sphere. Finally, the direction vector is scaled according to this distance and translated to the center of the sphere to obtain the sampling point located within the high-dimensional sphere. This operation is repeated for the entire batch to obtain a set of sampling embedding points that reflect semantic diversity. Based on this, we designed Matching Loss and Maximum Penetration Depth Loss, both of which are calculated based on Monte Carlo sampling points.

[0040] Specifically, the center coordinates of the visual sphere and the center vector of the text sphere are determined, and the radius is determined based on the visual uncertainty radius and the text uncertainty radius. The batch size, embedding dimension, and number of samples are set. An empty list is initialized, and the loop counter is set to 1. It is determined whether the loop counter is less than or equal to the batch size; if so, the center coordinates of the i-th sample are extracted from the center coordinates, and the radius value corresponding to the i-th sample is extracted from the radius. N d-dimensional direction vectors are sampled from the standard normal distribution to construct a direction matrix, and the L2 norm of each row vector is calculated to obtain the norm vector. The norm vector of each row of the direction matrix is ​​divided by its corresponding L2 norm. The process involves: obtaining a normalized direction matrix; sampling N scalar values ​​from a uniform distribution to obtain a uniform sampling vector; multiplying each element of the uniform sampling vector by its corresponding radius value to obtain a scaled radius vector; multiplying the scaled radius vector element-wise with the normalized direction matrix to obtain an offset vector relative to the center of the sphere; adding the offset vector element-wise with the center coordinates to obtain the final set of sampling points; adding the set of sampling points to a list; incrementing the loop counter by 1 until the loop counter is greater than the batch size; stacking all sampling points in the list along the batch dimension to construct the final sampling point tensor, thus enabling cross-modal image-text retrieval.

[0041] In this embodiment, the specific implementation process of the Monte Carlo sampling method for sphere embedding is as follows: Input: Batch size Embedded Dimensions Sphere center coordinates ,radius Number of samples N; Output: Set of sampling points ; 1) Initialize an empty list S to store all the sampled points; 2) Set up a loop counter =1; 3) Determine the loop counter Is it less than or equal to the batch size? If yes, proceed to step 4); otherwise, proceed to step 13). 4) From the center coordinates of the sphere Extract the first The center coordinates of each sample From the radius Extract the first The radius value corresponding to each sample ; 5) From the standard normal distribution Medium sampling indivual From the dimensional direction vector, we obtain the direction matrix. ,in express 3D identity matrix; 6) Calculate the direction matrix The norm vector is obtained by taking the L2 norm of each row vector in the vector. The vector contains A scalar value; 7) Direction matrix Divide each row vector by its corresponding L2 norm to perform a normalization operation, so that the length of each direction vector is 1, thereby projecting all direction vectors onto the unit sphere. 8) From the interval uniform distribution Medium sampling scalar values ​​are used to obtain a uniform sampling vector. ; 9) Convert the uniform sampling vector Each element in the matrix is ​​multiplied by its corresponding radius value. The scaled radius vector is obtained. This vector represents the actual distance from the center of the sphere to the sampling point; 10) Scale the radius vector With the normalized direction matrix Perform element-wise multiplication to obtain the offset vector P relative to the center of the sphere; 11) Relate the offset vector P to the coordinates of the sphere's center. By adding each element one by one, we obtain the final set of sampling points. This set contains N elements located at... Centered on Random sampling points within a sphere with radius ; 12) Set the sampling points Add it to list S, then add the loop counter. Increment by 1, return to step 3) and continue processing the next batch of samples; 13) Stack all the sample points in list S along the batch dimension to form the final sample point tensor. ; 14) Output sampling point tensor .

[0042] in, This represents a standard normal distribution with a mean of 0 and a covariance matrix equal to the identity matrix, also known as a multidimensional Gaussian distribution. It represents a uniform distribution, and its value range is the interval of real numbers greater than or equal to 0 and less than 1. Furthermore, this embodiment designs a matching loss and a maximum penetration depth loss, both of which are calculated based on Monte Carlo sampling points.

[0043] Among these, the matching loss is crucial. To ensure accurate semantic alignment between image-text pairs while preserving semantic uncertainty, we improve the matching loss in PCME. The original PCME method calculates the matching score by averaging the similarity of all sampled pairs. However, this averaging operation tends to dilute semantic uncertainty, especially when the semantic consistency of some sampled pairs is significantly higher than that of others. In contrast, our method selects the maximum matching score from all sampled pairs to emphasize the most semantically consistent correspondence between images and text, while still preserving potential diverse matching possibilities. This design allows the model to focus on the strongest semantic alignment, avoiding the loss of key discriminative information due to over-smoothing. For any image-text pair... The matching loss is defined as: ; ; In the above formula, This represents the matching loss function. This represents the maximum sigmoid similarity among all sampled pairs. , Representing the sampling in the image sphere and text sphere respectively. and One point, This represents the sigmoid function. , Indicates learnable parameters, Represents the L2 norm. This represents the number of samples for each ball. This indicates the magnitude of the text feature matching in the sampled image. , Features indicating the use of image text This represents conditional probability.

[0044] Where the matching probability Defined as the maximum sigmoid similarity among all sampled pairs; and The samples taken from the image sphere and the text sphere are respectively the first... and One point.

[0045] Maximum Penetration Depth Loss. To further enhance the model's ability to capture cross-modal semantic similarity while preserving the semantic uncertainty of each modality, we propose a novel maximum penetration depth loss. The core idea of ​​this loss function is to measure the extent to which a sampling point from one modality penetrates into the semantic region of another modality. Its goal is not only to assess cross-modal similarity but also to maintain the uniqueness and uncertainty boundaries of each modal semantic region. Specifically, by calculating the distance from each sampling point in one modality to the center of the corresponding instance sphere in the other modality, we can measure the degree of overlap between the semantic regions of the two modalities. This concept makes the representation of cross-modal interactions more nuanced: it focuses on the most aligned regions between the two modalities while also taking into account the variability and uncertainty within each modality. The loss is defined as follows: ; ; In the above formula, This represents the maximum penetration depth loss function. State of expression The ball One sampling point, , Representing modes The center and radius of the corresponding instance sphere, Indicates measurement from modality To mode Maximum penetration depth Indicates hyperparameters, This represents the number of samples for each ball. This represents the penetration depth from mode A to mode B. Indicates the first Each sampling feature.

[0046] in, This represents the k-th sampling point in the sphere representing mode A; and radius These represent the center and radius of the instance sphere corresponding to mode B, respectively; Measuring the maximum penetration depth from mode A to mode B; hyperparameters Control the sharpness of the penetration score (i.e., the sensitivity of the response).

[0047] Finally, the overall training loss of the model is composed of these two loss functions. and These factors are collectively determined and integrated into a unified optimization objective to jointly optimize multiple aspects of the model. The relative contribution of each loss is adjusted by a parameter. Control. This parameter serves as a key mechanism to balance the importance of semantic matching and preserving cross-modal uncertainty, ensuring that both aspects are optimized in a way that benefits the overall learning process. Therefore, the final training loss is expressed as: ; In the above formula, This indicates the adjustment parameter.

[0048] In summary, this embodiment proposes a novel High-Dimensional Ball Embedding (HDBE) framework: this framework can embed multimodal data into a unified space while explicitly preserving semantic uncertainty, thereby improving the accuracy of one-to-many image-text matching. Figure 3 As shown, unlike previous probabilistic or geometric representations, this embodiment models each modal instance as a high-dimensional sphere: the center is extracted by a modality-specific encoder, encoding the most semantically certain part of that modality; while the radius is determined through an attention-guided entropy estimation mechanism, naturally capturing semantic uncertainty and forming a confidence-aware region in the embedding space. This representation strategy has an isotropic and rotation-invariant structure, ensuring that the similarity metric remains consistent regardless of orientation in the latent space, thus faithfully approximating the inherent one-to-many characteristics in cross-modal semantics. Building on this representation, we further introduce a Monte Carlo sampling scheme to more effectively capture semantic diversity under uncertainty. Specifically, we sample points within each high-dimensional sphere to simulate diverse potential alignments between modalities. This random sampling not only enables the model to approximate all plausible correspondence possibilities in one-to-many text-to-image relationships but also provides a theoretical basis for designing a loss function based on sampled instances, thereby enhancing the framework's flexibility and robustness. Furthermore, to effectively measure cross-modal semantic relevance, we propose a novel maximum penetration depth loss to evaluate the extent to which a sampled point from one modality penetrates into the semantic region of another modality. This loss statement is intuitive, effective, and easy to integrate into existing retrieval frameworks.

[0049] Therefore, this embodiment differs from the prior art in the following technical features: 1) A novel High-Dimensional Sphere Embedding (HDBE) framework is proposed to model semantic uncertainty in a unified and isotropic embedding space. By representing cross-modal instances as high-dimensional spheres instead of points, this embodiment naturally captures the inherent diversity and ambiguity in one-to-many image-text matching scenarios, thereby improving robustness and retrieval accuracy.

[0050] 2) An end-to-end trainable architecture was designed to learn the center and radius of the semantic sphere through an attention-guided entropy mechanism; a Monte Carlo sampling strategy was further proposed, and a scalable semantic relevance loss based on maximum penetration depth was introduced to achieve more discriminative cross-modal matching.

[0051] Finally, the experiment is described in conjunction with the accompanying drawings of this embodiment: This embodiment conducts experiments on two widely used image-text retrieval benchmark datasets: CUB and MSCOCO. The CUB dataset contains 200 fine-grained bird species categories, and the highly detailed associations between images and text make it an ideal benchmark for evaluating cross-modal models. To maintain consistency with previous work (such as PCME), we used 150 categories for training and the remaining 50 for testing, ensuring a standard evaluation setup. CUB's fine-grained nature allows for a more nuanced evaluation of the model's ability to model complex semantic relationships between image features and text descriptions, making it particularly suitable for testing the effectiveness of retrieval models in handling fine semantic alignment. The MSCOCO dataset contains 123,287 images, each accompanied by 5 manually written text descriptions, providing rich multimodal data to support image-text retrieval tasks. Following standard protocols in recent image-text retrieval research, we used 113,287 images for training and reserved 5,000 images for validation. This dataset covers diverse and complex visual content, enabling comprehensive testing of the retrieval model's generalization ability.

[0052] During the evaluation, we report the results under two commonly used settings: 1K test set: Divide 5000 test images into 5 folds, take 1000 images for each test and take the average result, which helps to evaluate the stability of the model on different data subsets; Full 5K test set: Provides a more comprehensive assessment of the model's robustness.

[0053] Evaluation Metrics. Consistent with previous work, we use Recall@K as the primary evaluation metric to measure whether the model finds correct matches in the top K search results (K=1, 5). Specifically: R@1 represents the proportion of correct matches appearing first in the query results, reflecting the model's ability to retrieve the most relevant items; R@5 represents the proportion of correct matches appearing in the top five, reflecting the model's ability to capture relevant candidates within a broader search scope. In addition, we report R-Precision (RP) as a supplementary metric. Unlike R@K, which has a fixed K value, RP calculates the precision for each query across its top R results, where R is the number of true matches for that query. This adaptive metric provides a more nuanced assessment of the model's ability to accurately rank relevant candidates, particularly suitable for scenarios where the number of true matches varies depending on the query. To further evaluate the model's performance in "one-to-many" and semantically diverse matching scenarios, we also use the PMRP metric. PMRP considers both labeled matching and semantically plausible matching, providing a more comprehensive assessment of semantic alignment capabilities than relying solely on strictly labeled pairs.

[0054] The proposed HDBE method was implemented using PyTorch 2.1.0 and Python 3.8.12. The optimizer used was Adamp, with a learning rate of 0.0002. Hyperparameters for the visual and text modalities were also described. and Set to the same value: 0.01 on the MSCOCO dataset and 0.05 on the CUB dataset. The number of sampling points N within each high-dimensional sphere is set to 7. Loss balancing factor. On the MSCOCO and CUB datasets, the values ​​were set to 0.05 and 0.5, respectively. The remaining settings (such as embedding dimension and batch size) remained consistent with the previous PCME method.

[0055] For the results on the CUB dataset, zero-shot experiments were conducted on the CUB dataset, and the results are shown in Table 1. As shown in the table, our method achieves state-of-the-art performance on all metrics (RP and R@1). Compared with related methods (which use fan-shaped embeddings to model multimodal instances), our high-dimensional spherical embedding achieves a significant improvement in R@1: +2.3% and +0.4% for image-to-text (i2t), respectively. This indicates that in low-resource scenarios, the proposed spherical representation can more effectively capture semantic uncertainty and achieve more accurate cross-modal retrieval given a query. Furthermore, our method also outperforms the state-of-the-art in RP: +0.5% and +0.3% improvements in i2t and t2i, respectively, demonstrating that our method not only enhances fine-grained matching accuracy but also improves overall ranking quality and retrieval comprehensiveness.

[0056] Table 1. Comparison of retrieval performance on the cube with zero samples ; Here, i2t and t2i represent text retrieval and image retrieval, respectively.

[0057] Results on the MSCOCO dataset: We evaluated the model on the MSCOCO dataset with 1K and 5K test images, respectively, and reported PMRP, R@1, and R@5 metrics for image-to-text (i2t) and text-to-image (t2i) retrieval. On the 1K split (as shown in Table 2): HDBE achieved the best performance in the i2t task: R@1 = 70.4%, PMRP = 46.3%, with R@1 surpassing the current state-of-the-art method ESSE by 1.0 percentage point and PMRP matching it; meanwhile, R@5 reached 92.8%, the highest among all methods. Compared to PCME based on Gaussian distribution, HDBE improved R@1 by 1.6–2.4 percentage points, indicating that spherical embedding can more effectively model semantic uncertainty and improve top-1 retrieval accuracy. Compared to P2RM based on rectangular boxes, R@1 improved by approximately 3.8%, validating that isotropic spherical representations are more robust than axis-aligned rectangles in one-to-many matching. In the t2i task, HDBE also achieved the highest R@1 (57.0%), exceeding ESSE by 0.9 percentage points, and obtained the best PMRP (47.3%), indicating that HDBE achieved a good balance between semantic coverage and matching accuracy, and was also effective for reverse retrieval.

[0058] In the more challenging 5K partitioning: HDBE consistently outperforms all baseline methods in R@K (K=1,5), but slightly lags behind ESSE in PMRP. One possible reason is that HDBE focuses more on precisely matched samples, paying less attention to images or text that are similar in category but not strictly matched. For the i2t task, HDBE achieves R@1 = 46.3% and R@5 = 75.7%, with R@1 exceeding ESSE by 0.4 percentage points and P2RM by 2.2 percentage points, indicating that ball embedding can consistently improve model performance even in large-scale retrieval scenarios. For the t2i task, HDBE achieves the highest R@1 (33.9%), exceeding ESSE by 0.5 percentage points and P2RM by 2.4 percentage points, and R@5 (64.2%) is also slightly better than baseline methods. These results demonstrate that HDBE maintains a significant performance advantage under the more difficult 5K evaluation settings, validating the strong generalization ability and robustness of the proposed method.

[0059] Table 2 Performance Comparison Data of PMRP and R@K on MSCOCO

[0060] We conducted ablation experiments to verify the proposed high-dimensional sphere embedding (HDBE) and maximum penetration depth loss (…). The effectiveness of the ablation was assessed. The ablation results on the CUB and MSCOCO datasets are shown in Tables 3 and 4, respectively. For clarity, we use Models #1, #2, and #3 as examples: Model #1: Hyperparameters Set to 0 and remove That is, without using ball embedding, only using ball center (point embedding) for training and inference; Model #2: Reserved High-dimensional sphere embedding is used, but no dimensionality is introduced during training. ; Model #3: This is our complete model, which uses both sphere embedding and maximum penetration depth loss.

[0061] Table 3 Ablation Study Data on CUB ; Table 4 Ablation Study Data on MSCOCO ; The role of high-dimensional sphere embedding. As seen in Tables 3 (CUB zero-sample ablation) and 4 (MSCOCO ablation), the proposed high-dimensional sphere embedding plays a crucial role in improving the overall accuracy of image-text matching. Comparing Model #2 with Model #1 (the latter using only point representations), performance significantly decreased for both image-to-text (i2t) and text-to-image (t2i) retrieval. This indicates that relying solely on deterministic point embeddings cannot effectively capture the inherent semantic uncertainty in one-to-many correspondences. To further validate the effectiveness of HDBE, we additionally reported the R@5 metric in our CUB ablation experiments. The results show that replacing the sphere embedding with point embedding significantly reduced R@5: a 0.9% decrease for text retrieval and a 0.6% decrease for image retrieval. This consistent decrease across multiple metrics fully demonstrates that using high-dimensional spheres can encode richer semantic variations, thereby maintaining robust retrieval performance.

[0062] The effect of maximum penetration depth loss. To evaluate... To assess the contributions, we compared Model #3 (including) on ​​CUB (Table 3) and MSCOCO (Table 4). ) and Model #2 (excluding) ).

[0063] In the CUB zero-sample experiment: For i2t retrieval, introduce R@1 increased from 50.1% to 51.0% (+0.9%), and RP increased from 27.3 to 27.6 (+0.3). For t2i retrieval, R@1 increased from 37.5% to 38.2% (+0.7%), and RP increased from 27.6 to 28.1 (+0.5).

[0064] Regarding the 1K partitioning of MSCOCO: i2t: R@1 changed from 69.3% to 70.4% (+1.1%), RP changed from 45.8 to 46.3 (+0.5); t2i: R@1 changed from 56.5% to 57.0% (+0.5%), and RP changed from 47.0 to 47.3 (+0.3).

[0065] On the more challenging 5K division: i2t: R@1 changed from 45.8% to 46.3% (+0.5%), RP changed from 34.6 to 34.8 (+0.2%). t2i: R@1 changed from 33.5% to 33.9% (+0.4%), and RP changed from 35.2 to 35.3 (+0.1%).

[0066] Although R@5 is relatively stable, the continuous improvement of R@1 and RP indicates that: It primarily enhances the ability to identify the most relevant matches and improves overall ranking consistency.

[0067] This gain stems from The core design principle is that it measures the maximum penetration depth of a sample point from one modality into the semantic region of another modality, emphasizing the most semantically representative match rather than averaging all sample pairs. This mechanism directly strengthens top-1 alignment, thus significantly improving R@1, and enhances ranking stability (reflected in RP improvement) by regularizing the embedding space.

[0068] To investigate the robustness of the proposed method, we conducted parameter sensitivity experiments on the CUB dataset, systematically tuning key hyperparameters to examine their impact on retrieval performance. Specifically, we focus on analyzing the effects of the following two hyperparameters. (set up ): The weight of the sphere radius (i.e., semantic uncertainty) in the embedded representation; : Controlling the maximum penetration depth loss ( We also analyzed the gradient weighting in the total loss; furthermore, we analyzed the impact of the number of samples N on the experimental results.

[0069] The impact. Figure 5and Figure 6 Showing different Performance at a given value. When The model performance is most stable during this period. In terms of the RP metric, both image-to-text (i2t) and text-to-image (t2i) retrieval peak within this range; when... At that time, performance dropped sharply. On R@1, the model... More sensitive: when At that time, the R@1 of i2t decreased from over 50% to approximately 35%. These results indicate that a proper balance needs to be struck between the semantic center and the radius of uncertainty: if If the value is too small, its role in uncertainty modeling is weak; if... If the value is too large, it will overwhelm the precise semantic alignment signal, leading to a decrease in top-1 retrieval performance.

[0070] The impact. Figure 7 and Figure 8 Showing Impact on performance. RP in When it reaches its optimal state, Both excessively small and excessively large values ​​lead to performance degradation. This reflects the trade-off between emphasizing the "most penetrating match point" and overall ranking consistency in the penetration depth loss: Too small will lead to The discrimination signal is weak, and the model degenerates into average matching; A value that is too large can lead to over-focusing on individual sampling points, disrupting the overall sorting structure. The R@1 curve shows a similar trend, while R@5 is relatively stable, indicating... It primarily enhances fine-grained top-1 alignment without compromising candidate diversity.

[0071] The effect of the number of samples N. Figure 9 and Figure 10 This study demonstrates the impact of the number of samples, N, on the RP and R@K metrics for cross-modal retrieval on the CUB dataset. Interestingly, the model achieves optimal performance when N=7. Intuitively, one might expect that increasing the number of data samples would continuously improve performance; however, our experiments reveal a counterintuitive trend: performance initially increases with N, but then decreases. We attribute this phenomenon to the zero-shot nature of the CUB retrieval task: as N increases, the model is more prone to overfitting to classes seen during training, thus impairing its generalization ability to unseen classes. This explanation is consistent with the overall trend of the curves—performance initially increases and then decreases with increasing N. Furthermore, this finding underscores the importance of a careful trade-off between sampling diversity and semantic noise: oversampling can introduce redundant or misleading information, interfering with the learning of transferable representations.

[0072] Furthermore, for the visualization analysis of HDBE, to intuitively demonstrate the effectiveness of our proposed High-Dimensional Ball Embedding, we use t-SNE to visualize the multimodal representations learned by the model. For each sample, we sample 40 feature points. Figure 11 Visualizations of the high-dimensional embedding space projected onto a two-dimensional plane for 12 image categories and 9 text categories are presented. The results clearly demonstrate that sphere-based embeddings generate highly separable clusters: even for visually very similar bird species, the model effectively captures fine-grained inter-class differences and maintains clear category separation in the high-dimensional space. Furthermore, the text embeddings exhibit well-preserved semantic structure—different linguistic features are clearly clustered into their respective clusters. These observations demonstrate that our method can simultaneously learn discriminative representations for both visual and textual modalities, highlighting its ability to model fine-grained semantic differences across categories. Moreover, the elliptical distribution of the mapped points empirically validates the consistency and stability of our sampling strategy.

[0073] like Figure 9 The image shows a T-SNE visualization of the high-dimensional sphere embeddings. The learned visual and text embeddings are located on 12 and 9 subclasses of the CUB dataset, respectively. For each image and caption, we sampled 40 instance embeddings.

[0074] In summary, this embodiment proposes a High-Dimensional Ball Embedding (HDBE) method for intelligent image-text retrieval in surveillance systems, aiming to better model the inherent semantic uncertainty in one-to-many matching scenarios. Unlike existing geometric embedding methods that rely on orientation alignment, HDBE utilizes the isotropy and rotation invariance of a high-dimensional sphere to represent each modality instance as a high-dimensional sphere—where the center encodes the most reliable semantic information, and the radius adaptively reflects semantic uncertainty through an attention-guided entropy mechanism. Building upon this, we introduce a Monte Carlo sampling strategy to explore semantic diversity within the sphere and design a novel Maximum Penetration Depth Loss to enhance the matching of the most aligned regions across modalities. Extensive experiments demonstrate that the proposed method outperforms current mainstream methods on both the CUB and MSCOCO benchmarks, validating its effectiveness in improving retrieval accuracy and robustness. Despite the significant advantages of HDBE in one-to-many image-text retrieval tasks, future work can be expanded in the following directions: extending the framework to other multimodal tasks (such as video-text alignment, cross-modal generation, etc.); exploring more efficient sampling strategies or deterministic approximation methods to reduce computational overhead; and further studying the fusion mechanism of ball embeddings and large-scale pre-trained models (such as CLIP) to improve generalization ability in open domain scenarios.

[0075] Reference Figure 2 An image-text cross-modal retrieval system based on high-dimensional sphere embedding includes: The first module 201 is used to extract features from the target image and target text respectively through the backbone network and word embedding method to obtain visual features and text features; The second module 202 is used to perform center calculation of sphere embedding and semantic uncertainty modeling on visual features and text features respectively based on the sphere encoder, to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius; The third module 203 is used to perform similarity learning on the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius through the Monte Carlo sampling method of sphere embedding, so as to realize cross-modal retrieval of image and text.

[0076] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0077] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image-text cross-modal retrieval method based on high-dimensional sphere embedding, characterized in that, Includes the following steps: Based on the target image and target text, feature extraction is performed on the target image and target text respectively through backbone network and word embedding methods to obtain visual features and text features; Based on the ball encoder, the center calculation of sphere embedding and semantic uncertainty modeling are performed on visual features and text features respectively, to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius; By employing a Monte Carlo sampling method based on sphere embedding to learn the similarity between the visual sphere center vector, the visual uncertainty radius, the text sphere center vector, and the text uncertainty radius, cross-modal image-text retrieval is achieved.

2. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 1, characterized in that, The step of calculating the center of the sphere embedding and modeling the semantic uncertainty of visual and text features based on the sphere encoder to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector, and text uncertainty radius specifically includes: Based on the visual sphere encoder, the visual features are used to perform center calculation of sphere embedding and semantic uncertainty modeling through an attention-guided entropy mechanism, resulting in the visual sphere center vector and visual uncertainty radius. Based on the text ball encoder, the text features are embedded into a ball and the semantic uncertainty is modeled by an attention-guided entropy mechanism to obtain the text ball center vector and the text uncertainty radius.

3. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 2, characterized in that, The step of calculating the center of the visual sphere and modeling semantic uncertainty by performing sphere embedding on visual features through an attention-guided entropy mechanism, based on a visual sphere encoder, to obtain the visual sphere center vector and the visual uncertainty radius, specifically includes: The visual features are sequentially subjected to fully connected transformation and global average pooling extraction to obtain the global semantic information of the visual features and construct the first component. Dimension transformation is performed on deep features in visual features to obtain the dimensionally transformed deep features; After the dimensionality transformation, the deep features are activated by hyperbolic tangent and then the attention vectors are generated through a fully connected layer. The attention weights of each visual category feature are obtained by normalization through the Softmax function. The deep features after dimensionality transformation are multiplied by the attention weights and then activated by the Sigmoid activation function to generate the second component; By combining the first and second components, a visual sphere center vector is constructed; The semantic uncertainty radius is obtained by measuring the information entropy of the attention weight distribution.

4. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 3, characterized in that, The specific expression for calculating the center of sphere embedding of visual features using the attention-guided entropy mechanism is as follows: ； ； ； ； In the above formula, Indicates the first component. Indicates the second component, Indicates a fully connected transformation. Indicates global average pooling. Indicates visual characteristics, Represents the attention vector. This represents the Softmax function. This represents the hyperbolic tangent activation function. Represents the dimensional transformation function. This represents the center vector of the visual sphere.

5. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 4, characterized in that, The specific expression for measuring semantic uncertainty based on the information entropy of the attention weight distribution is as follows: ； In the above formula, Represents the radius of visual uncertainty. This represents a hyperparameter that modulates the intensity of visual semantic uncertainty. This represents the attention weights in the attention vector corresponding to the visual features. Indicates the radius.

6. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 5, characterized in that, The step of obtaining the text sphere center vector and text uncertainty radius by performing center calculation and semantic uncertainty modeling of text features based on the text sphere encoder through an attention-guided entropy mechanism specifically includes: Contextual modeling of text features is performed using a bidirectional GRU network to capture bidirectional semantic information. Attention features are extracted from text features using two fully connected layers and the activation function tanh, generating word-level attention distribution weights. The text features are multiplied by the word-level attention distribution weights, and then weighted features are generated using the Sigmoid activation function. By combining bidirectional semantic information and weighted features, a text sphere center vector is constructed; The semantic uncertainty radius is obtained by measuring the information entropy of the distribution of word-level attention weights.

7. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 6, characterized in that, The step of performing similarity learning on the visual sphere center vector, visual uncertainty radius, text sphere center vector, and text uncertainty radius using the Monte Carlo sampling method with sphere embedding to achieve cross-modal image-text retrieval specifically includes: Determine the center coordinates of the spheres based on the visual sphere center vector and the text sphere center vector, determine the radius based on the visual uncertainty radius and the text uncertainty radius, set the batch size, embedding dimension and sampling number, initialize an empty list and set the loop counter to 1; Determine if the loop counter is less than or equal to the batch size. If so, extract the center coordinates of the i-th sample from the sphere center coordinates and extract the radius value corresponding to the i-th sample from the radius. Sample N d-dimensional direction vectors from the standard normal distribution, construct a direction matrix, and calculate the L2 norm of each row vector to obtain the norm vector; Divide the norm vector of each row of the direction matrix by its corresponding L2 norm to obtain the normalized direction matrix; Sample N scalar values ​​from a uniform distribution to obtain a uniform sampling vector, and multiply each element of the uniform sampling vector by the corresponding radius value to obtain a scaled radius vector; The scaled radius vector is multiplied element-wise with the normalized direction matrix to obtain the offset vector relative to the center of the sphere. The offset vector is added element by element to the center coordinates of the sphere to obtain the final set of sampling points. The set of sampling points is then added to a list, and the loop counter is incremented by 1 until the loop counter is greater than the batch size. All sampling points in the list are stacked along the batch dimension to construct the final sampling point tensor, enabling cross-modal retrieval of image text.

8. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 7, characterized in that, The loss function of the sphere embedding Monte Carlo sampling method includes matching loss and maximum penetration depth loss, wherein the expression for the matching loss is as follows: ； ； In the above formula, This represents the matching loss function. This represents the maximum sigmoid similarity among all sampled pairs. , Representing the sampling in the image sphere and text sphere respectively. and One point, This represents the sigmoid function. , Indicates learnable parameters, Represents the L2 norm. This represents the number of samples for each ball. This indicates the magnitude of the text feature matching in the sampled image. , Features indicating the use of image text This represents conditional probability.

9. The image-text cross-modal retrieval method based on high-dimensional sphere embedding according to claim 8, characterized in that, The expression for the maximum penetration depth loss is as follows: ； ； In the above formula, This represents the maximum penetration depth loss function. State of expression The ball One sampling point, , Representing modes The center and radius of the corresponding instance sphere, Indicates measurement from modality To mode Maximum penetration depth Indicates hyperparameters, This represents the number of samples for each ball. This represents the penetration depth from mode A to mode B. Indicates the first Each sampling feature.

10. An image-text cross-modal retrieval system based on high-dimensional sphere embedding, characterized in that, Includes the following modules: The first module is used to extract features from the target image and target text respectively through the backbone network and word embedding method to obtain visual features and text features. The second module is used to perform center calculation and semantic uncertainty modeling of sphere embedding on visual features and text features respectively, based on the sphere encoder, to obtain the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius; The third module is used to perform similarity learning on the visual sphere center vector, visual uncertainty radius, text sphere center vector and text uncertainty radius through the Monte Carlo sampling method of sphere embedding, so as to realize cross-modal retrieval of images and text.

Images