Multi-modal hash retrieval method and device for body agent

Abstract

The invention provides a multi-modal Hash retrieval method and device for an agent with a body, and relates to the technical field of artificial intelligence application, and the method comprises the following steps: constructing a multi-modal Hash retrieval model for the agent with the body; the multi-modal Hash retrieval model comprises a multi-modal feature encoder and a Kolmogorov-Arnod Hash layer which are connected in sequence; constructing a joint loss function fusing multi-modal supervision optimal transmission alignment loss and self-paced learning loss, and training the multi-modal Hash retrieval model by using the joint loss function to obtain a trained multi-modal Hash retrieval model; receiving multi-modal feature retrieval data, inputting the multi-modal feature retrieval data into the trained multi-modal hash retrieval model, and outputting a hash code of the multi-modal feature retrieval data; and calculating the Hamming distance of the Hash code of the multi-modal feature retrieval data, and returning a multi-modal feature retrieval result according to a sorting result of the Hamming distance. According to the method, the retrieval effect and the retrieval efficiency can be effectively improved.

Description

Technical Field

[0001] This invention relates to the technical field of artificial intelligence applications, and in particular to a multimodal hash retrieval method and apparatus for embodied intelligent agents. Background Technology

[0002] In recent years, with the breakthrough development of embodied intelligence technology, the interaction between robot entities and the physical environment has generated massive amounts of heterogeneous perceptual data. Information from multimodal sensors, including vision, force, hearing, and position sensors, is growing exponentially, constructing a complex embodied experience data space. Faced with this high-dimensional, cross-modal perceptual information flow, embodied agents require real-time and efficient cross-modal retrieval capabilities to support their environmental understanding, task planning, and behavioral decision-making.

[0003] Multimodal hash retrieval technology is an important means of multimodal retrieval. Existing multimodal hash retrieval technologies are mainly divided into two categories: cross-modal hash retrieval and hash manifold learning. The basic goal of cross-modal hash retrieval is to learn compact binary hash codes and achieve semantic alignment and fast search across different modalities through unified representation learning. In the past, it was usually combined with hand-designed features and linear projection strategies. In recent years, deep learning methods have been used for text and image modal feature extraction and contrastive learning methods for positive and negative sample matching for final loss calculation. Although deep hashing methods have achieved significant performance in cross-modal image and text retrieval, the linear projection plus binary encoding paradigm has encountered a key bottleneck. Similarly, it faces more severe challenges in embodied intelligence scenarios, with higher requirements for computing power allocation and real-time performance. Dimensionality reduction through linear mapping usually leads to a large loss of semantic information during feature manifold approximation, resulting in poor retrieval performance and increased model complexity, which contradicts the core goal of lightweight hash retrieval. Hash manifold learning typically employs linear dimensionality reduction to encode high-dimensional data into a compact binary representation. This inevitably leads to geometric distortion of the underlying data manifold during the projection from the original feature space to the Hamming space. Cross-modal retrieval techniques that combine manifold structure modeling and binary hash encoding aim to generate compact binary hash codes by preserving the geometric structure of the data on the low-dimensional manifold, thereby achieving efficient similarity search. Recent methods have used graph learning for manifold fitting. While graph-based methods exhibit excellent topology preservation capabilities through neighborhood learning, their computational complexity reaches O(n³) for large datasets, resulting in low retrieval efficiency and fundamentally limiting their scalability in practical applications. Summary of the Invention

[0004] To address the problems of poor retrieval results and low retrieval efficiency in the existing technologies, this invention proposes a multimodal hash retrieval method and apparatus for embodied intelligent agents, which can effectively improve retrieval results and efficiency.

[0005] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A multimodal hash retrieval method for embodied intelligent agents includes the following steps: S1. Construct a multimodal hash retrieval model for an embodied intelligent agent, wherein the multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence; S2. Construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and use the joint loss function to train the multimodal hash retrieval model to obtain a trained multimodal hash retrieval model; S3. Receive the multimodal feature retrieval data of the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data; S4. Calculate the Hamming distance of the hash codes of the multimodal feature retrieval data, and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

[0006] Preferably, the multimodal feature encoder includes an image feature encoder for extracting latent semantic representations of images and a text feature encoder for extracting latent semantic representations of text, wherein the calculation expression for the latent semantic representations of images is as follows:

[0007] in, For latent semantic representation of images, For image feature encoders, This serves as the input for the image feature encoder; The computational expression for the latent semantic representation of the text is as follows:

[0008] in, For latent semantic representation of text, For text feature encoders, This is the input for the text feature encoder.

[0009] Preferably, the Kolmogorov-Arnold hash layer receives the image latent semantic representation and the text latent semantic representation, and outputs the hash codes of the image latent semantic representation and the text latent semantic representation, respectively; The hash code for the latent semantic representation of the image is calculated as follows:

[0010] The hash code for the latent semantic representation of the text is calculated as follows:

[0011] in, The hash code for the latent semantic representation of the image. The hash code is the latent semantic representation of the text. For the order function mapping of hash, The output of the hash's sequential function mapping, This is the activation function.

[0012] Preferably, the calculation expression for the joint loss function is as follows:

[0013] in, For the joint loss function, For self-learning loss, For multimodal supervision of optimal transmission alignment loss, These are the hyperparameters for the optimal transport alignment loss and self-stepping learning loss in multimodal supervision.

[0014] Preferably, the multimodal supervised optimal transmission alignment loss The calculation expression is as follows:

[0015] in, For batch size, The hash code of the latent semantic representation of text i A hash code, The hash code of the latent semantic representation of an image j A hash code, For positive sample alignment scores, Alignment score for negative samples; Positive sample alignment score The calculation expression is as follows:

[0016] in, For row-by-row normalization factor, for transpose, As the discriminant factor; The negative sample alignment score The calculation expression is as follows:

[0017] in, For the estimated false negative rate, The total number of negative samples for each batch. For the optimal transfer matrix, This is the negative sample masking function.

[0018] Preferably, the optimal transmission matrix The calculation expression is as follows:

[0019] in, Let be the transition matrix. For the cost matrix, The first in the transition matrix i Line 1 j Column elements; The cost matrix The calculation expression is as follows:

[0020] in, This is a subtraction operation for broadcast.

[0021] Preferably, the self-stepping learning loss The calculation expression is as follows:

[0022] in, For the self-synchronization loss of text pairs, For the autosynchronization loss of image pairs, The step-by-step loss of the text and image pairs.

[0023] The present invention also proposes a multimodal hash retrieval device for embodied intelligent agents, comprising: A multimodal hash retrieval model building unit is used to build a multimodal hash retrieval model for an embodied intelligent agent. The multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence. The training unit is used to construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and to train the multimodal hash retrieval model using the joint loss function to obtain a trained multimodal hash retrieval model. The output unit is used to receive multimodal feature retrieval data from the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data. The retrieval unit is used to calculate the Hamming distance of the hash codes of the multimodal feature retrieval data and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

[0024] The present invention also proposes a computer device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform operations as described in the multimodal hash retrieval method for embodied agents.

[0025] The present invention also proposes a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform operations as described in the multimodal hash retrieval method for embodied intelligent agents.

[0026] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a multimodal hash retrieval method and apparatus for embodied agents. Firstly, the multimodal hash retrieval method constructs a model architecture that integrates a multimodal feature encoder and a Kolmogorov-Arnold hash layer, namely, a multimodal hash retrieval model for embodied agents. It then introduces a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss for training, achieving efficient cross-modal hashing dimensionality reduction and manifold preservation, effectively improving the alignment accuracy and retrieval performance of cross-modal data. Simultaneously, based on the combination of optimal transmission theory and self-stepping learning strategy, the adaptability of the multimodal hash retrieval model to complex modal interactions is enhanced. Furthermore, the multimodal feature retrieval data of the embodied agent is input into the trained multimodal hash retrieval model, which outputs the hash codes of the multimodal feature retrieval data. The Hamming distance of the hash codes of the multimodal feature retrieval data is then calculated, and the multimodal feature retrieval results are returned based on the ranking of the Hamming distances, significantly improving the accuracy and efficiency of multimodal retrieval. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a multimodal hash retrieval method for embodied intelligent agents proposed in an embodiment of the present invention. Figure 2 This is another flowchart illustrating a multimodal hash retrieval method for embodied intelligent agents proposed in this embodiment of the invention; Figure 3 This is a diagram showing the comparison of KAOTH image-text training encoding time in the embodiments of the present invention; Figure 4 This diagram illustrates the TopN retrieval accuracy of KAOTH as proposed in this embodiment of the invention. Figure 5 This diagram illustrates the recall and retrieval accuracy of KAOTH as proposed in this embodiment of the invention. Figure 6 This diagram illustrates the t-SNE effect of the KAOTH method proposed in this embodiment of the invention compared to other methods. Figure 7This diagram illustrates the effect of the loss fusion hyperparameter α proposed in this embodiment of the invention on model performance. Figure 8 This diagram illustrates the structure of a multimodal hash retrieval device for embodied intelligent agents, as proposed in an embodiment of the present invention. Figure 9 This is a schematic diagram of the structure of a computer device proposed in an embodiment of the present invention; 91. Processor; 92. Memory; 93. Communication interface; 94. Communication bus; 95. Executable instructions. Detailed Implementation

[0028] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings; The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] Example 1 like Figure 1 and Figure 2 As shown, this embodiment proposes a multimodal hash retrieval method for embodied agents. This method is optimized based on a cross-modal retrieval paradigm using dual pre-trained encoders. Its structure fits the alignment framework in the current embodied application scenario. The structure includes a Kolmogorov-Arnold hash layer proposed for manifold hash dimensionality reduction encoding optimization of the original hash layer, and a manifold alignment self-stepping optimal transmission loss proposed for the loss function. It includes modules for dual-modal supervised optimal transmission alignment and self-stepping learning, specifically including the following steps: S1. Construct a multimodal hash retrieval model for an embodied intelligent agent, wherein the multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence; To effectively extract features from both image and text modalities, the multimodal feature encoder in this embodiment includes an image feature encoder for extracting latent semantic representations of images and a text feature encoder for extracting latent semantic representations of text. The image feature encoder integrates multi-scale latent semantic representations of images through cascaded multi-head attention modules and multilayer perceptron projection. The text feature encoder uses a CLIP architecture with multi-head attention and implements location-aware context label aggregation. Through this dual-branch encoding path, the original multimodal input... The computational expression for nonlinear mapping to the latent semantic representation of the image is as follows:

[0030] in, For latent semantic representation of images, For image feature encoders, This serves as the input for the image feature encoder; The computational expression for the latent semantic representation of the text is as follows:

[0031] in, For latent semantic representation of text, For text feature encoders, This serves as the input for the text feature encoder; The Kolmogorov-Arnold hash layer is a bimodal Kolmogorov-Arnold hash layer (KAHL) based on manifold projection. First, the nonlinear mapping function... Defined as a B-spline basis function, this function is a piecewise polynomial function defined on a set of nodes:

[0032] in These are the weights of the spline function. silu Represents the activation function, coefficients These are learnable parameters, composed of a combination of univariate basis functions. Recursion is defined as follows:

[0033] Where the spline order This determines the continuity of the basis functions and the length of the support interval; secondly, once the basic building blocks of the nonlinear mapping function are established... These basis functions can then be systematically arranged into a matrix structure consisting of univariate activation functions:

[0034] in, Indicates the first l Layer j The node and the first l +1 floor i A weighted combination of univariate basis functions among nodes. and The number of nodes in each consecutive layer is specified. Therefore, KAHL is defined as a sequential function mapping of hashes, as follows:

[0035] in .

[0036] The Kolmogorov-Arnold hash layer receives the image latent semantic representation and the text latent semantic representation, and outputs the hash codes of the image latent semantic representation and the text latent semantic representation, respectively; Combining the calculation expression of the piecewise polynomial function and the sequential function mapping of hashing, this embodiment defines the calculation expression of the hash code of the latent semantic representation of the image as follows:

[0037] The hash code for the latent semantic representation of the text is calculated as follows:

[0038] in, The hash code is the latent semantic representation of the image, representing a continuous relaxed binary encoding of the image features. A hash code representing the latent semantic representation of text, representing a continuous, relaxed binary encoding of text features. For the order function mapping of hash, KAHL By using B-spline basis functions, the underlying manifold structure of the data is effectively captured while maintaining parameter efficiency. The output of the hash's sequential function mapping, This is the activation function.

[0039] S2. Construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and use the joint loss function to train the multimodal hash retrieval model to obtain a trained multimodal hash retrieval model; The calculation expression for the joint loss function is as follows:

[0040] in, For the joint loss function, For self-learning loss, For multimodal supervision of optimal transmission alignment loss, The hyperparameters for multimodal supervised optimal transport alignment loss and self-stepping learning loss; hyperparameters By maintaining the global distribution geometry through optimal transmission and enhancing local manifold consistency through geometry-aware hard negative sample mining, this loss function, which combines self-stepping learning loss with dual-supervised transmission loss, ensures that alignment maintains both the topological stability of the embedding space and the fidelity of semantic correspondence, thereby effectively preventing overfitting noise interactions and optimizing inter-manifold transmission.

[0041] Given aligned text-image pairs The multimodal supervised optimal transmission alignment loss The calculation expression is as follows:

[0042] in, The batch size represents the superposition operation of the modal matrices. The hash code of the latent semantic representation of text i A hash code, Let j be the j-th hash code in the hash code of the latent semantic representation of the image. For positive sample alignment scores, Alignment score for negative samples; Based on the InfoNCE loss, the positive samples are aligned with the scores. The computation expression is summarized as follows:

[0043] in, This is a row-wise normalization factor used to maintain numerical stability. for transpose, As the discriminant factor; Sharpness is determined by temperature scaling control.

[0044] This invention addresses the false negative sample problem in contrastive learning by assuming a proportion of true negative samples. This invention uses an optimal transfer matrix. Redefine negative samples, the negative sample alignment score The calculation expression is as follows:

[0045] in, For the estimated false negative rate, The total number of negative samples for each batch. For the optimal transfer matrix, This is the negative sample masking function.

[0046] This invention views multimodal supervised optimal transport alignment as a manifold space mapping framework, aiming to simultaneously model intermodal correlations and intramodal interactions through cross-modal data matching. To construct sample pairs, this invention defines and optimizes positive and negative samples based on a distance metric between text and image modalities. The Sinkhorn algorithm is used to calculate the optimal transport scheme, generating a transport matrix to quantify cross-modal distribution differences. This formula utilizes an optimal transport cost metric to measure text distribution. and image distribution The differences between them can be identified, thereby enabling more effective cross-modal alignment by explicitly modeling the distribution relationships.

[0047] In the optimal transfer learning optimization process, the most important definition is the optimal transfer matrix. Its progressively increasing weights represent the discrimination and selection of difficult and easy negative samples, adjusting the positive correlation of samples in cross-view alignment and reducing the model's fit to easy samples. Therefore, the optimal transfer matrix is ​​defined as follows. The calculation expression is as follows:

[0048] in, Let be the transition matrix. Cost matrix The similarity between samples is explicitly modeled by constructing the squared Euclidean distance between cross-modal features, using this geometry-based metric within the view. This will determine the difficulty of transmission (the higher the cost, the more difficult the transmission). Let be the element in the i-th row and j-th column of the transition matrix; The cost matrix The calculation expression is as follows:

[0049] in, This is a subtraction operation for broadcast.

[0050] This invention defines the transition matrix as E, uses the Sinkhorn algorithm to determine the weights of negative samples, iterates through entropy regularization to solve the optimal transition problem, and assigns transition weights to complex and simple samples. For cross-modal problems, the transition matrix of cross-modal global text-image pairs is measured. Its definition is:

[0051]

[0052] in The transfer matrix is ​​obtained through dual variables. u and v Iterative updates are performed; both variables are all-one matrices, with the same size as the image and text matrices. Regularization coefficients. It achieves a combined effect by balancing sparse optimal transport solutions and entropy smoothing. The ever-increasing... This causes the solutions to move away from the sparse distribution that easily gets trapped in local minima, and tend towards a uniform distribution, thus implicitly balancing cooperation and controlling [the situation]. Sinkhorn The trade-off between efficiency and optimality in iteration.

[0053] The optimal transmission matrix The scalar operations in the computational expressions are extended to set-valued operations. This invention defines probability distribution pairs. Optimal transition matrix operator This operator acts on the entire probability distribution and can be formally represented as:

[0054] Entropy-based optimal transmission method Evaluation text and images The degree of alignment of the distributions. However, its basic version T has a key limitation: the combined distance matrix. It blends two different types of discrepancies—semantic differences between different categories across modalities, and natural differences within features of the same category. This blending makes... It is difficult to distinguish irrelevant samples during alignment calculation. To address this issue, this invention introduces a label-guided alignment method to separate these two types of differences during distance calculation:

[0055] in, This indicates an indicator function used to offset the transportation costs between cross-modal sample pairs sharing a common label. ,in Controlling the suppression intensity. This operation explicitly protects the class-discriminative semantics from variations in intra-class sample hardness, thus preserving the geometric hierarchy.

[0056] Although passed While adaptive supervision effectively isolates inter-class interference, it is insufficient to maintain intra-modal structural consistency. To address this limitation while preserving the integrity of overall semantic information, this invention derives an improved optimal transport operator through supervised manifold unwrapping and intra- / inter-modal equalization. Its formal definition is:

[0057] The first item Cross-modal mass transit under controlled label-guided orthogonal subspace projection explicitly decouples modal-specific topological dependencies by imposing geometric orthogonality between text and visual embeddings. Simultaneously, it provides intramodal optimal transport supervision. and It anchors intramodal discriminability, preserves the essential geometry of each distribution, and filters out spurious intermodal interactions. These mechanisms are unified through entropy-regularized Wasserstein alignment. It dynamically balances intermodal transferability and intramodal stability, ensuring cross-modal distance distribution, evolving sample distance correspondence, and semantic consistency.

[0058] The self-stepping learning loss The calculation expression is as follows:

[0059] in, For the self-synchronization loss of text pairs, For the autosynchronization loss of image pairs, The step-by-step loss of the text and image pairs.

[0060] The effectiveness of autostep learning has been validated in cross-modal tasks. Introducing autostep learning into multimodal autonomous learning strategies enables the systematic evaluation of cross-modal relation pairs. By distinguishing between simple and challenging relation pairs, the algorithm overcomes the limitation of uniform response strategies in adapting to complex modal interactions, ultimately improving algorithm efficiency. For cross-modal sample pairs, the autostep loss for image pairs... The text portion is defined as follows:

[0061]

[0062]

[0063] in Indicates and Semantically related subsets Is with Semantically independent subsets It's a temperature parameter. (Auto-pacing factor) Defined as:

[0064] in Defined as the cosine similarity to the sample pair. Define different modal characteristics of the input computation. It is training dynamic parameter control.

[0065] S3. Receive the multimodal feature retrieval data from the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data. Specifically, unlike the training process, the calculation expression for the hash code of the visual latent semantic representation in the inference process is as follows:

[0066] The hash code for the latent semantic representation of the text is calculated as follows:

[0067] in, A hash code for visual latent semantic representation. The hash code is the latent semantic representation of the text. For the order function mapping of hash, For activation function, and These are the binary hash codes of the final outputs of image and text reasoning, respectively; S4. Calculate the Hamming distance of the hash codes of the multimodal feature retrieval data, and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

[0068] In embodied intelligent systems, such as service robots or autonomous driving platforms, cross-modal data often manifests as embodied scene image modalities (e.g., environmental visual images) and command modalities (e.g., user language commands). The high-dimensional nonlinear structure of this data easily leads to low geometric fitting efficiency and semantic interference. This embodiment proposes a multimodal hash retrieval method for embodied intelligent agents to address the problems of low efficiency in manifold geometric approximation and difficulty in decoupling high-order cross-modal correlations in nonlinear dimensionality reduction. It uses node-parameterized B-spline basis functions for local nonlinear dimensionality reduction, achieving efficient approximation of the manifold geometry, effective decoupling of high-order cross-modal semantic correlations, and maintaining the topological stability of the data manifold differential. To address the local optimum trap in cross-modal alignment and the semantic bias caused by random negative sampling in contrastive learning, it uses the self-stepping optimal transmission loss of manifold alignment. This achieves the effects of maintaining the topological consistency of the B-spline manifold by minimizing the Wasserstein distance, adaptively prioritizing reliable sample pairs with weights, and eliminating intermodal interference based on geometrically aware hard negative sampling.

[0069] This embodiment also constructs a model architecture that integrates a multimodal feature encoder and a Kolmogorov-Arnold hash layer, and introduces a joint loss function that integrates multimodal supervised optimal transport alignment loss and self-stepping learning loss for training. This achieves efficient cross-modal hash dimensionality reduction and manifold preservation, effectively improving the alignment accuracy and retrieval performance of cross-modal data. At the same time, based on the combination of optimal transport theory and self-stepping learning strategy, the adaptability of the multimodal hash retrieval model to complex modal interactions is enhanced. Furthermore, the multimodal feature retrieval data is input into the trained multimodal hash retrieval model, the hash code of the multimodal feature retrieval data is output, the Hamming distance of the hash code of the multimodal feature retrieval data is calculated, and the multimodal feature retrieval result is returned according to the sorting result of the Hamming distance, which significantly improves the accuracy and efficiency of multimodal retrieval.

[0070] Example 2 This embodiment further illustrates the multimodal hash retrieval method for embodied intelligent agents proposed in the above embodiments. Although cross-modal hashing has become a promising method for effectively retrieving heterogeneous multimodal data, especially suitable for scenarios where embodied intelligent systems need to integrate vision, language, proprioception, and motion perception, and has great potential, existing methods still face two basic limitations: geometric distortion of the feature manifold caused by the linear projection layer and excessively high computational complexity related to graph-based alignment strategies. Therefore, the multimodal hash retrieval method for embodied agents proposed in the above embodiments is a large-scale cross-modal hash retrieval method based on the Kolmogorov-Arnold representation theorem and optimal transmission, denoted as KAOTH. The core innovation of KAOTH lies in two complementary components: First, the Kolmogorov-Arnold hash layer uses adaptive B-spline basis combination to nonlinearly encode the feature geometry, mitigating semantic collapse through parameter-efficient function approximation while maintaining the spatiotemporal continuity of the motion trajectory; Second, the manifold-aligned self-stepping optimal transmission loss function constructs the coupling matrix of embodied perception constraints, realizing cross-modal manifold mapping through optimal quality transmission between feature distributions and dynamically adapting to environmental changes.

[0071] The multimodal hash retrieval method for embodied agents proposed in the above embodiments further integrates three aspects of embodied intelligence optimization: 1) Optimization of the cost matrix for monitoring signal conditioning ensures intra-class information preservation and cross-modal consistency; 2) Proprioceptive feedback-guided manifold calibration maintains physical feasibility; 3) The real-time hash indexing mechanism supports millisecond-level action decisions. In embodied tasks such as robot operation and navigation, KAOTH significantly improves the accuracy and efficiency of multimodal understanding and decision-making, while reducing computational complexity to O(nlogn), meeting the needs of embedded deployment.

[0072] See Figure 3 Comparing the training and encoding times of the KAOTH method in this embodiment with other methods, it was found that the training time of the KAOTH method is longer than that of the AGAH, DADH, and SCAHN methods, but shorter than that of the DCHMT and DHaPH methods; the encoding time of the KAOTH method is the same as that of the SCAHN method, and shorter than that of other methods.

[0073] See Figure 4 By comparing the TopN image retrieval accuracy of the KAOTH method in this embodiment with that of other methods, it was found that the TopN image retrieval accuracy of the KAOTH method is higher than that of other methods. See Figure 5By comparing the recall and retrieval accuracy of the KAOTH method in this embodiment with those of other methods, it was found that the recall and retrieval accuracy of the KAOTH method is higher than that of other methods. See Figure 6 By comparing the t-SNE effect of the KAOTH method in this embodiment with that of other methods, it was found that the t-SNE effect of the KAOTH method is better than that of other methods.

[0074] This embodiment also experimentally verifies the multimodal hash retrieval method for embodied intelligent agents proposed in the above embodiments, as shown in Tables 1 and 2. Figure 7 Based on experimental verification, the KAOTH framework proposed in this invention has the following beneficial effects: In terms of cross-modal retrieval performance, through B-spline manifold parameterization of KAHL and optimal transmission alignment of MASPOT, it achieves 87.06% mAP for 128-bit I2T retrieval on the MIRFLICKR-25K dataset (surpassing the baseline method DHaPH by 2.13%), 77.60% mAP for T2I task on MSCOCO (leading by 5.77%), and 65.23% mAP for 16-bit T2I task on IAPR TC-12 (better than DNcH by 4.78%). In terms of bit length robustness, the manifold stabilization mechanism achieves monotonic performance improvement across the entire bit segment (16→128-bit mAP gain: I2T +2.16%, T2I +1.56%), with mAP fluctuation ≤0.2% at 128-bit, and 0.82% higher performance in 16-bit supervised than unsupervised retrieval. In terms of computational efficiency, the training time is 0.85 hours (37% shorter than DHaPH), the inference speed is 17ms / sample (on par with lightweight AGAH), and the dynamic bit expansion architecture achieves 87.02% mAP with 170.61M parameters. In terms of architecture universality, the integrated CLIP architecture improves mAP by an average of 1.21% (DHaPH+KAHL achieves +1.63%), the ViT-B32 backbone achieves 86.07% mAP (FLOPs are only 4.41G), and Pareto optimality is achieved when the hyperparameter α=0.2.

[0075] Table 1 Experimental results of the KAOTH framework on various datasets.

[0076] Table 2 Generalization experiments of KAHL on different models

[0077] Example 3 See Figure 8 This embodiment proposes a multimodal hash retrieval device, including: A multimodal hash retrieval model building unit is used to build a multimodal hash retrieval model for an embodied intelligent agent. The multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence. The training unit is used to construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and to train the multimodal hash retrieval model using the joint loss function to obtain a trained multimodal hash retrieval model. The output unit is used to receive multimodal feature retrieval data from the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data. The retrieval unit is used to calculate the Hamming distance of the hash codes of the multimodal feature retrieval data and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

[0078] In this embodiment, the multimodal hash retrieval method for embodied agents first constructs a model architecture that integrates a multimodal feature encoder and a Kolmogorov-Arnold hash layer. It then introduces a joint loss function that integrates multimodal supervised optimal transport alignment loss and self-stepping learning loss for training, achieving efficient cross-modal hash dimensionality reduction and manifold preservation, effectively improving the alignment accuracy and retrieval performance of cross-modal data. Simultaneously, based on the combination of optimal transport theory and self-stepping learning strategy, the adaptability of the multimodal hash retrieval model to complex modal interactions is enhanced. Furthermore, the multimodal feature retrieval data is input into the trained multimodal hash retrieval model, outputting the hash codes of the multimodal feature retrieval data. The Hamming distance of the hash codes of the multimodal feature retrieval data is then calculated, and the multimodal feature retrieval results are returned based on the ranking of the Hamming distances, significantly improving the accuracy and efficiency of multimodal retrieval.

[0079] Example 4 This embodiment presents a computer device, see [link to documentation]. Figure 9 It includes: a processor 91, a memory 92, a communication interface 93 and a communication bus 94, wherein the processor 91, the memory 92 and the communication interface 93 communicate with each other through the communication bus 94; The processor 91, memory 92, and communication interface 93 communicate with each other via a communication bus 94. The communication interface 93 is used for network communication with other devices, such as clients or other servers. The processor 91 executes executable instructions 95, specifically performing the operations of the multimodal hash retrieval method for embodied intelligent agents. Specifically, the executable instructions 95 may include program code. The processor 91 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0080] Memory 92 is used to store executable instructions 95. Memory 92 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0081] Executable instruction 95 can be invoked by processor 91 to cause the computer device to perform the following operations: S1. Construct a multimodal hash retrieval model for an embodied intelligent agent, wherein the multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence; S2. Construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and use the joint loss function to train the multimodal hash retrieval model to obtain a trained multimodal hash retrieval model; S3. Receive the multimodal feature retrieval data of the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data; S4. Calculate the Hamming distance of the hash codes of the multimodal feature retrieval data, and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

[0082] In this embodiment, the multimodal hash retrieval method first constructs a model architecture that integrates a multimodal feature encoder and a Kolmogorov-Arnold hash layer, namely, an embodied agent multimodal hash retrieval model. It then introduces a joint loss function that integrates multimodal supervised optimal transport alignment loss and self-stepping learning loss for training, achieving efficient cross-modal hash dimensionality reduction and manifold preservation, effectively improving the alignment accuracy and retrieval performance of cross-modal data. Simultaneously, based on the combination of optimal transport theory and self-stepping learning strategy, the adaptability of the multimodal hash retrieval model to complex modal interactions is enhanced. Furthermore, the multimodal feature retrieval data of the embodied agent is input into the trained multimodal hash retrieval model, which outputs the hash code of the multimodal feature retrieval data. The Hamming distance of the hash code of the multimodal feature retrieval data is then calculated, and the multimodal feature retrieval result is returned based on the ranking result of the Hamming distance, significantly improving the accuracy and efficiency of multimodal retrieval.

[0083] This embodiment also proposes a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on a computer device, it causes the computer device to perform the operation of the multimodal hash retrieval method for embodied intelligent agents, specifically including the following steps: S1. Construct a multimodal hash retrieval model for an embodied intelligent agent, wherein the multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence; S2. Construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and use the joint loss function to train the multimodal hash retrieval model to obtain a trained multimodal hash retrieval model; S3. Receive the multimodal feature retrieval data of the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data; S4. Calculate the Hamming distance of the hash codes of the multimodal feature retrieval data, and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

[0084] In this embodiment, the multimodal hash retrieval method first constructs a model architecture that integrates a multimodal feature encoder and a Kolmogorov-Arnold hash layer, namely, an embodied agent multimodal hash retrieval model. It then introduces a joint loss function that integrates multimodal supervised optimal transport alignment loss and self-stepping learning loss for training, achieving efficient cross-modal hash dimensionality reduction and manifold preservation, effectively improving the alignment accuracy and retrieval performance of cross-modal data. Simultaneously, based on the combination of optimal transport theory and self-stepping learning strategy, the adaptability of the multimodal hash retrieval model to complex modal interactions is enhanced. Furthermore, the multimodal feature retrieval data of the embodied agent is input into the trained multimodal hash retrieval model, which outputs the hash code of the multimodal feature retrieval data. The Hamming distance of the hash code of the multimodal feature retrieval data is then calculated, and the multimodal feature retrieval result is returned based on the ranking result of the Hamming distance, significantly improving the accuracy and efficiency of multimodal retrieval.

[0085] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A multimodal hash retrieval method for embodied intelligent agents, characterized in that, Includes the following steps: S1. Construct a multimodal hash retrieval model for an embodied intelligent agent, wherein the multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence; S2. Construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and use the joint loss function to train the multimodal hash retrieval model to obtain a trained multimodal hash retrieval model; S3. Receive the multimodal feature retrieval data of the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data; S4. Calculate the Hamming distance of the hash codes of the multimodal feature retrieval data, and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

2. The multimodal hash retrieval method for embodied intelligent agents according to claim 1, characterized in that, The multimodal feature encoder includes an image feature encoder for extracting latent semantic representations of images and a text feature encoder for extracting latent semantic representations of text. The calculation expression for the latent semantic representations of images is as follows: in, For latent semantic representation of images, For image feature encoders, This serves as the input for the image feature encoder; The computational expression for the latent semantic representation of the text is as follows: in, For latent semantic representation of text, For text feature encoders, This is the input for the text feature encoder.

3. The multimodal hash retrieval method for embodied intelligent agents according to claim 2, characterized in that, The Kolmogorov-Arnold hash layer receives the image latent semantic representation and the text latent semantic representation, and outputs the hash codes of the image latent semantic representation and the text latent semantic representation, respectively; The hash code for the latent semantic representation of the image is calculated as follows: The hash code for the latent semantic representation of the text is calculated as follows: in, The hash code for the latent semantic representation of the image. The hash code is the latent semantic representation of the text. For the order function mapping of hash, The output of the hash's sequential function mapping, This is the activation function.

4. The multimodal hash retrieval method for embodied intelligent agents according to claim 3, characterized in that, The calculation expression for the joint loss function is as follows: in, For the joint loss function, For self-learning loss, For multimodal supervision of optimal transmission alignment loss, These are the hyperparameters for the optimal transport alignment loss and self-stepping learning loss in multimodal supervision.

5. The multimodal hash retrieval method for embodied intelligent agents according to claim 4, characterized in that, The multimodal supervised optimal transmission alignment loss The calculation expression is as follows: in, For batch size, The hash code of the latent semantic representation of text i A hash code, The hash code of the latent semantic representation of an image j A hash code, For positive sample alignment scores, Alignment score for negative samples; Positive sample alignment score The calculation expression is as follows: in, For row-by-row normalization factor, for transpose, As the discriminant factor; The negative sample alignment score The calculation expression is as follows: in, For the estimated false negative rate, The total number of negative samples for each batch. For the optimal transfer matrix, This is the negative sample masking function.

6. The multimodal hash retrieval method for embodied intelligent agents according to claim 5, characterized in that, The optimal transmission matrix The calculation expression is as follows: in, Let be the transition matrix. For the cost matrix, The first in the transition matrix i Line 1 j Column elements; The cost matrix The calculation expression is as follows: in, This is a subtraction operation for broadcast.

7. The multimodal hash retrieval method for embodied intelligent agents according to claim 5, characterized in that, The self-stepping learning loss The calculation expression is as follows: in, For the self-synchronization loss of text pairs, For the autosynchronization loss of image pairs, The auto-stepping loss of the text-image pair.

8. A multimodal hash retrieval device for embodied intelligent agents, characterized in that, include: A multimodal hash retrieval model building unit is used to build a multimodal hash retrieval model for an embodied intelligent agent. The multimodal hash retrieval model includes a multimodal feature encoder and a Kolmogorov-Arnold hash layer connected in sequence. The training unit is used to construct a joint loss function that integrates multimodal supervised optimal transmission alignment loss and self-stepping learning loss, and to train the multimodal hash retrieval model using the joint loss function to obtain a trained multimodal hash retrieval model. The output unit is used to receive multimodal feature retrieval data from the embodied intelligent agent, input the multimodal feature retrieval data into the trained multimodal hash retrieval model, and output the hash code of the multimodal feature retrieval data. The retrieval unit is used to calculate the Hamming distance of the hash codes of the multimodal feature retrieval data and return the multimodal feature retrieval results based on the sorting results of the Hamming distance.

9. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the multimodal hash retrieval method for embodied agents as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the multimodal hash retrieval method for embodied intelligent agents as described in any one of claims 1-8.

Images