A multi-modal embodied intelligent data retrieval method based on dynamic spatio-temporal alignment and view decoupling

Abstract

The application discloses a kind of multi-modal embodied intelligent data retrieval method based on dynamic space-time alignment and view decoupling, for the space-time heterogeneous of multi-modal data in embodied intelligent scene, semantic gap and storage efficiency problem, propose solution: by adaptive time window alignment algorithm, according to sampling rate and task urgency synchronizes space-time, output time alignment confidence quantification solution quality;Adopt two-way Transform to extract space-time characteristics, use gate mechanism to generate joint representation by confidence adaptive fusion;Through orthogonal and diversity constraint decoupling intra-modal semantics and cross-modal association, construct double view;Based on hybrid expert framework, combined with scene type and confidence dynamically activate expert subset to realize quantitative compression;Use hierarchical index distributed vector library to support double view joint retrieval, through active learning driven knowledge update and migration.The data management efficiency, retrieval accuracy and robustness can be improved, to provide support for real-time perception-decision closed loop.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to the fields of embodied intelligence and multimodal data management, and particularly to a multimodal embodied intelligence data retrieval method based on dynamic spatiotemporal alignment and view decoupling. Background Technology

[0002] With the rapid development of embodied intelligence technology, robots need to process heterogeneous data from multiple sensors such as vision, speech, touch, and force when performing tasks in complex environments. Existing multimodal data management methods mainly suffer from the following technical bottlenecks: First, there is the issue of spatiotemporal heterogeneity. In dynamic environments, there are significant differences in the acquisition frequency and temporal characteristics of data from different modalities. For example, although the invention patent CN118966161A discloses a method for constructing a general artificial spatiotemporal intelligence large model, it proposes a multi-dimensional object abstraction and spatiotemporal coding method. However, this scheme is designed for the construction of a general large model and does not take into account the characteristics of large differences in the sampling frequency of heterogeneous sensors in embodied intelligence scenarios (such as 100Hz for touch and 30Hz for vision) and high requirements for temporal synchronization. It lacks dynamic time window alignment and confidence quantification mechanisms, resulting in low accuracy of multimodal data fusion. This spatiotemporal offset makes it difficult to effectively fuse and align multimodal data in a unified spatiotemporal coordinate system.

[0003] Secondly, there is the semantic gap problem. Visual, speech, and tactile data exhibit significant differences in semantic representation. The representation of the same physical event can be completely different in different modalities. Establishing cross-modal semantic associations is a core challenge in current research. Furthermore, existing multimodal representation methods often employ single-view encoding. For example, the knowledge graph-based multimodal content understanding method disclosed in invention patent CN120372538A, while introducing a spatiotemporal perceptual graph attention network, lacks a decoupling mechanism between the semantic view and the aligned view. This makes cross-modal semantic associations susceptible to intramodal noise interference, limiting retrieval accuracy.

[0004] Third, there is the issue of storage efficiency. Embodied intelligent systems need to process massive amounts of multimodal data in real time. Traditional dense vector representation methods suffer from high storage overhead and low retrieval efficiency, making it difficult to meet the needs of robots' real-time perception-decision closed loop.

[0005] Fourth, there is the issue of environmental adaptability. When embodied intelligent systems operate in complex and dynamic environments, they need to dynamically adjust their data representation strategies according to environmental changes and task requirements, but existing methods lack flexible adaptive mechanisms.

[0006] Therefore, there is an urgent need for a multimodal data retrieval method with dynamic spatiotemporal alignment and view decoupling for embodied intelligence scenarios, in order to address the shortcomings of the existing technologies. Summary of the Invention

[0007] Based on this, the present invention provides a multimodal embodied intelligence data retrieval method based on dynamic spatiotemporal alignment and view decoupling, in order to solve the problems of multimodal spatiotemporal heterogeneity, semantic gap, storage efficiency and environmental adaptability faced by existing technologies in embodied intelligence scenarios.

[0008] The present invention adopts the following technical solution: A multimodal embodied intelligent data retrieval method based on dynamic spatiotemporal alignment and view decoupling, the method comprising: The multimodal data acquisition and spatiotemporal synchronization steps involve real-time acquisition of heterogeneous multimodal time-series data of embodied intelligent devices operating in the physical environment. Through an adaptive time window alignment algorithm, data with different sampling frequencies are mapped to a unified time reference coordinate system based on the dynamic fluctuations of the sensor's real-time sampling rate and the urgency of the task. The quantified time alignment confidence is then output to quantify the alignment quality. The dynamic spatiotemporal awareness cross-modal representation learning step uses a dual-channel Transformer encoder to process the temporal and spatial features of the heterogeneous multimodal time series data respectively. The temporal and spatial features are adaptively weighted and fused through a gating mechanism to generate a joint representation containing spatiotemporal context-aware features. The hierarchical semantic-aligned view decoupling step involves using a semantic view encoder to extract fine-grained semantic information within the modality from the joint representation to construct a semantic view, and using an alignment view encoder to extract cross-modal correlations between task instructions and skill data to construct an alignment view. Orthogonal constraints and diversity constraints are used to ensure that the feature spaces of the semantic view and the alignment view are independent of each other, forming a dual-view joint representation. The scene-adaptive sparse representation and quantization compression steps involve constructing a hybrid expert sparse coding framework. Based on the scene type, task semantic features, and the time alignment confidence, the relevant expert subset in the hybrid expert sparse coding framework is dynamically activated to generate sparse feature representations and perform quantization compression. Efficient storage and retrieval steps for distributed vector databases; constructing a distributed vector database with a hierarchical index structure; supporting semantic-aligned dual-view joint retrieval. The knowledge update and transfer steps driven by active learning use an uncertainty-driven sample selection strategy to screen high-uncertainty samples for expert annotation. The annotation results are then combined with a cross-scenario knowledge transfer framework to update model parameters, achieving dynamic adaptation of the source and target domain distributions and incremental model updates without forgetting previous knowledge.

[0009] As a further improvement of the present invention, the quantized time alignment confidence is determined in the following manner: In the adaptive time window alignment algorithm, cubic spline interpolation is used for low-frequency modes and uniform downsampling is used for high-frequency modes. Cross-modal time correlation is established through the time alignment factor matrix. Calculate the time alignment residual. When the time alignment residual exceeds a preset threshold, mark the time alignment confidence as low and trigger a fault tolerance mechanism. This fault tolerance mechanism adjusts the weight allocation strategy of subsequent feature extraction modules to compensate for time-series deviations. Wherein, the time alignment confidence of the quantization is f∈[0,1].

[0010] As a further improvement of the present invention, in the dual-channel Transformer encoder, the temporal encoder adopts the Temporal Attention mechanism to model temporal dependencies in order to capture the dynamic evolution characteristics of multimodal data in the time dimension; the spatial encoder adopts the Cross-View Attention mechanism to model cross-view spatial associations in order to capture the geometric structure and spatial layout characteristics of multimodal data in the spatial dimension. The gating mechanism dynamically adjusts the fusion weights based on the time alignment confidence level, including: increasing the weight ratio of spatial dimension features when the time alignment confidence level is low; and increasing the weight ratio of time dimension features when the time alignment confidence level is high.

[0011] As a further improvement of the present invention, in the dynamic spatiotemporal perception cross-modal representation learning step, cross-modal alignment is performed at the semantic level, instance level and pixel level through a multi-granularity semantic alignment mechanism, respectively using the contrastive learning loss function, triplet ranking loss function and dense matching loss function; Through the multi-granularity semantic alignment mechanism, cross-modal semantic gradient associations from coarse-grained to fine-grained are formed in the feature space.

[0012] As a further improvement of the present invention, the hybrid expert sparse coding framework includes 64 expert networks, each expert being responsible for feature representation in different semantic domains, including motion scenes, natural scenes, man-made object scenes, and interactive scenes.

[0013] The scene-adaptive sparse representation and quantization compression steps specifically include: Construct a scene classifier and a task encoder to output scene type distribution and task semantic features in real time; The scene type distribution, the task semantic features, and the time alignment confidence are fused in multiple dimensions to generate expert activation weights. Feature calculation is performed only on the expert subset whose weight exceeds a preset threshold. For scenarios with low time alignment confidence, the expert subset with noise resistance and fault tolerance is activated first to generate sparse feature representations.

[0014] As a further improvement of the present invention, in the hierarchical index structure, the top layer uses an inverted index for coarse-grained classification by scene type and task domain, and the bottom layer uses product quantization to achieve fine-grained vector retrieval. The hierarchical index structure supports dynamic updates, enabling the online insertion of newly collected multimodal data and synchronous updates to the relevant index structure. The dynamic update adaptively adjusts the segmentation threshold of the top-level index based on the new sample distribution output by the knowledge update and transfer steps driven by active learning.

[0015] As a further improvement of the present invention, the semantic-aligned dual-view joint retrieval performs approximate nearest neighbor search in the semantic view and the alignment view respectively, and obtains the final retrieval result through adaptive weighted fusion, with the fusion weight dynamically adjusted according to the query type.

[0016] As a further improvement of the present invention, the uncertainty-driven sample selection strategy predicts uncertainty through a Monte Carlo Dropout estimation model and selects samples with high uncertainty for expert annotation.

[0017] As a further improvement of the present invention, the cross-scenario knowledge transfer framework includes: Reduce the distribution differences between the source and target domains by using adversarial domain adaptation methods; A continuous learning mechanism is introduced, and a flexible weight solidification method is used to apply regularization constraints to the key parameters in the model; The regularization intensity of the elastic weight solidification method is dynamically adjusted according to the time alignment confidence. Specifically, when the time alignment confidence is low, the regularization constraints on key parameters are increased to protect learned knowledge; when the time alignment confidence is high, the regularization constraints are decreased to accelerate the model's adaptation to new scenarios.

[0018] As a further improvement of the present invention, the method further includes: The query rewriting and expansion steps involve semantic analysis of the original query before retrieval to generate multiple related query vectors. Potentially related concepts are extracted from the knowledge base using semantic clustering methods to generate extended query vectors; The search results of the extended query vector are fused with the search results of the original query vector using a confidence-weighted method to expand the search coverage and improve the search accuracy of complex commands.

[0019] The technical solutions provided by the embodiments of the present invention have the following beneficial effects: 1. This invention uses an adaptive time window alignment algorithm combined with a time alignment confidence quantification mechanism to dynamically compensate for sensor sampling fluctuations and differences in task urgency, effectively solving the temporal deviation problem caused by static alignment of general spatiotemporal large models. It automatically switches the spatial feature-dominated mode in low confidence scenarios, significantly improving spatiotemporal alignment accuracy and system robustness. 2. This invention adopts a hierarchical semantic-aligned view decoupling design, which overcomes the limitations of single-view representation through orthogonal constraints and multi-granularity semantic alignment, and effectively bridges the cross-modal semantic gap; 3. This invention innovatively introduces a hybrid expert sparse coding framework, which combines a multi-dimensional activation strategy based on scene type, task semantics, and alignment confidence, along with a hierarchical index structure that supports dynamic updates. Compared with static expert solutions, this significantly reduces computational redundancy and achieves a dual breakthrough in storage and retrieval efficiency. 4. This invention forms a "data quality-model update" closed loop by actively learning-driven uncertain sample screening and elastic weight solidification regularization dynamic adjustment, which solves the catastrophic forgetting problem when migrating across scenarios, greatly enhances environmental adaptability, and provides highly reliable data support for the embodied intelligence real-time perception-decision closed loop. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall architecture of the multimodal embodied intelligent data retrieval method of the present invention; Figure 2 This is a detailed flowchart illustrating the multimodal embodied intelligent data retrieval method of the present invention; Figure 3 This is a schematic diagram of the structure of the dynamic spatiotemporal alignment module provided in an embodiment of the present invention; Figure 4 This is an architecture diagram of hierarchical semantic-aligned view decoupling provided in an embodiment of the present invention; Figure 5 MoE framework diagram for scene adaptive sparse coding provided in embodiments of the present invention; Figure 6 This is an architecture diagram of a distributed retrieval system provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the working principle of the multimodal embodied intelligent data retrieval method provided in this embodiment of the invention; Detailed Implementation

[0021] The technical solution of the present invention will be clearly and specifically described below with reference to the accompanying drawings and specific embodiments. It will be understood by those skilled in the art that the specific embodiments described herein are only for explaining the relevant invention and not for limiting the invention. The described embodiments are merely a part of the embodiments of the present invention, and for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0022] like Figures 1 to 6 As shown, an embodiment of the present invention provides a multimodal embodied intelligent data retrieval method based on dynamic spatiotemporal alignment and view decoupling, which may include the following steps: Step S1: Multimodal data acquisition and spatiotemporal synchronization Specifically, it collects heterogeneous multimodal time-series data of embodied intelligent devices operating in the physical environment in real time. Through an adaptive time window alignment algorithm, it maps data with different sampling frequencies to a unified time reference coordinate system based on the dynamic fluctuation of the sensor's real-time sampling rate and the urgency of the task, and outputs quantified time alignment confidence to quantify the alignment quality.

[0023] Furthermore, the quantized time alignment confidence is determined in the following way: The adaptive time window alignment algorithm specifically includes: based on the real-time sampling frequency of each modal data, using cubic spline interpolation to push up the time of low-frequency modal data, and using uniform downsampling to shrink the time of high-frequency modal data; constructing a time alignment factor matrix and establishing cross-modal time correlation; Calculate the time alignment residual. When the time alignment residual exceeds a preset threshold, mark the time alignment confidence as low and trigger a fault tolerance mechanism. This fault tolerance mechanism adjusts the weight allocation strategy of subsequent feature extraction modules to compensate for time-series deviations. Wherein, the time alignment confidence of the quantization is f∈[0,1].

[0024] like Figure 3 As shown, the specific steps are as follows: S101. Collect visual, speech, tactile, depth, and inertial measurement unit data through a multi-sensor fusion framework to construct a raw multimodal data stream; among which, visual data is collected by an RGB camera at 30Hz, tactile data is collected by a distributed tactile array at 100Hz, speech data is sampled by a microphone array at 16kHz, depth data is collected by a ToF sensor at 15Hz, and inertial data is sampled by an IMU at 200Hz.

[0025] S102. Design an adaptive time window alignment algorithm to dynamically interpolate and resample based on the timestamps of each modality data, and map heterogeneous time series data to a unified time reference coordinate system; use cubic spline interpolation for low-frequency modes and uniform downsampling for high-frequency modes, and establish cross-modal time correlation through the time alignment factor matrix.

[0026] During the execution of the adaptive time window alignment algorithm, the time alignment residual is further calculated: by comparing the mean squared error (MSE) or cross-correlation coefficient of the original sampling points and the interpolated / resampled points, the residual is normalized to the 0-1 interval to obtain the time alignment confidence (denoted as f, f∈[0,1]). When f<0.8, it is marked as low time alignment confidence data, triggering the fault tolerance mechanism: the subsequent feature extraction module automatically reduces the weight ratio of time dimension features and prioritizes the use of spatial dimension features (such as depth, location, and geometric structure) to compensate for feature distortion caused by temporal deviation.

[0027] S103. Construct a spatial coordinate transformation module to uniformly calibrate the spatial position and attitude parameters of different sensors. Establish a global spatial coordinate system through hand-eye calibration and sensor extrinsic parameter estimation to achieve spatial alignment of multimodal data.

[0028] Step S2: Cross-modal representation learning with dynamic spatiotemporal awareness Specifically, a dual-channel Transformer encoder is used to process the temporal and spatial features of the heterogeneous multimodal time-series data respectively. The temporal and spatial features are adaptively weighted and fused through a gating mechanism to generate a joint representation that includes spatiotemporal context-aware features.

[0029] Furthermore, the gating mechanism dynamically adjusts the fusion weights based on the time alignment confidence level: when the time alignment confidence level is low, the weight ratio of spatial dimension features is increased; when the time alignment confidence level is high, the weight ratio of time dimension features is increased.

[0030] The specific steps are as follows: S201. Design a dual-channel Transformer encoder architecture to handle feature extraction in the temporal and spatial dimensions respectively. The temporal encoder uses the Temporal Attention mechanism to model temporal dependencies in order to capture the dynamic evolution features of multimodal data in the temporal dimension. The spatial encoder uses the Cross-View Attention mechanism to model cross-view spatial relationships in order to capture the geometric structure and spatial layout features of multimodal data in the spatial dimension.

[0031] S202. Construct a dynamic spatiotemporal context awareness module to dynamically adjust the fusion weights of time and space features based on the current task type and environmental state; introduce task indication vectors and scene embedding vectors, and achieve adaptive spatiotemporal feature weighted fusion through a gating mechanism.

[0032] The gating mechanism is specifically implemented as follows: receiving the time-aligned confidence score f from S102, and dynamically adjusting the fusion weight w of the time dimension features and the spatial dimension features. t (Time weight) and ws (Spatial weights), satisfying w t +w s =1. When f≥0.8, increase the time weight (w) t =0.7, w s =0.3), making full use of time series information to capture dynamic evolution characteristics; when f < 0.8, reduce the time weight (w t =0.3, w s =0.7), using the stability of spatial information to compensate for timing errors.

[0033] S203. A multi-granularity semantic alignment mechanism is proposed, which performs cross-modal alignment at the semantic level, instance level, and pixel level respectively. The semantic level adopts a contrastive learning loss function, the instance level adopts a triplet ranking loss function, and the pixel level adopts a dense matching loss function to achieve cross-modal semantic association from coarse to fine.

[0034] Specifically, multi-granularity semantic alignment mechanisms include: A contrastive learning loss function is used at the semantic level to achieve semantic alignment between text instructions and the global visual scene; At the instance level, a triplet sorting loss function is used to align a specific target in the instruction with a specific instance in the visual scene. A dense matching loss function is used at the pixel level to align instruction details with pixel-level features in the visual scene; Through the multi-granularity semantic alignment mechanism, cross-modal semantic gradient associations from coarse-grained to fine-grained are formed in the feature space.

[0035] Step S3: Hierarchical semantics - View alignment decoupling Specifically, a semantic view is constructed by extracting fine-grained semantic information within the modality from the joint representation using a semantic view encoder, and an aligned view is constructed by extracting cross-modal correlations between task instructions and skill data using an aligned view encoder. Orthogonal constraints and diversity constraints are used to ensure that the feature spaces of the semantic view and the aligned view are independent of each other, thus forming a dual-view joint representation.

[0036] like Figure 4 As shown, the specific steps are as follows: S301. Design a semantic view encoder that focuses on extracting fine-grained semantic information within a modality, including low-level features such as object attributes, spatial layout, and motion patterns. Employ a hierarchical attention mechanism to capture local texture and edge features at a shallow level and aggregate global semantic information at a deep level.

[0037] S302. Design an aligned view encoder that focuses on modeling the cross-modal relationship between task instructions and skill data; introduce a cross-modal contrastive learning framework to achieve semantic alignment between task requirements and knowledge items by maximizing the mutual information between task text and corresponding skill visual representations.

[0038] S303. Construct a view decoupling regularization loss, which ensures that the feature spaces of the semantic view and the aligned view are independent of each other through orthogonal constraints, avoids the degradation of feature representation through diversity constraints, and ensures the aggregation of similar samples in the feature space through compactness constraints.

[0039] Specifically, the orthogonal constraint loss function is L_orthogonal=Σ‖V_semantic·V_alignment^T‖_F^2, and the diversity constraint loss function is L_diversity=-Σ‖V_semantic-V_alignment‖_F^2, where V_semantic represents the semantic view feature matrix and V_alignment represents the alignment view feature matrix.

[0040] Step S4: Scene-Adaptive Sparse Representation and Quantization Compression Specifically, a hybrid expert sparse coding framework (MoE framework) is constructed. Based on the current scene type, task semantic features, and the time alignment confidence, relevant expert subsets in the hybrid expert sparse coding framework (MoE framework) are dynamically activated to generate sparse feature representations and perform quantization compression.

[0041] Furthermore, the scene-adaptive sparse representation and quantization compression steps include: Construct a scene classifier and a task encoder to output scene type distribution and task semantic features in real time; The scene type distribution, the task semantic features, and the time alignment confidence are fused in multiple dimensions to generate expert activation weights. Feature calculation is performed only on the expert subset whose weight exceeds a preset threshold. For scenarios with low time alignment confidence, the expert subset with noise resistance and fault tolerance is activated first to generate sparse feature representations.

[0042] like Figure 5 As shown, the specific steps are as follows: S401. Design a scene-aware hybrid expert sparse coding framework (MoE framework). Construct a hybrid expert sparse coding framework (MoE framework) containing multiple experts. Each expert is responsible for feature representation in different semantic domains, including motion scenes, natural scenes, artificial object scenes, and interactive scenes. The total number of experts is set to 64, and each expert is an independent fully connected network.

[0043] S402. Introduce a scene classifier and a task encoder to dynamically activate relevant expert subsets based on the current environment type and task objective; the scene classifier outputs the scene type distribution, the task encoder extracts task semantic features, and the two are fused to generate expert activation weights.

[0044] The generation of expert activation weights employs a multi-dimensional fusion strategy: the scene type distribution output by the scene classifier (e.g., motion scenes, natural scenes, man-made object scenes, interactive scenes), the task semantic features extracted by the task encoder (e.g., grasping, carrying, assembling), and the time-aligned confidence score f are concatenated, and expert activation weights are generated through a fully connected layer. Specifically, when f < 0.8, a subset of experts with noise resistance and fault tolerance (e.g., motion scene experts, interactive scene experts) is activated first, while time-sensitive experts (e.g., natural scene experts) are avoided to prevent retrieval errors caused by time-series biases.

[0045] It should be noted that, through extensive experimental verification, when the normalized value of the time alignment residual is less than 0.2 (i.e., f < 0.8), the impact of the temporal deviation of the subsequent feature extraction module on the retrieval accuracy exceeds the acceptable range. Therefore, 0.8 is set as the threshold.

[0046] S403. Vector quantization is used to compress and encode sparse representations, mapping high-dimensional floating-point vectors to low-dimensional integer codebooks; an improved Gumbel-Softmax quantizer is used to maintain gradient propagation during the training phase and achieve efficient table lookup during the inference phase.

[0047] S404. By jointly optimizing the sparse coding framework through multi-objective loss functions, the quantization error is minimized, semantic fidelity is maintained, and the discriminative ability is enhanced, while ensuring the balance of expert activation.

[0048] Step S5: Efficient Storage and Retrieval of Distributed Vector Databases Construct a distributed vector database with a hierarchical index structure to support semantic-aligned dual-view joint retrieval.

[0049] Specifically, a distributed vector database with a hierarchical index structure is constructed. The hierarchical index structure supports approximate nearest neighbor search on the semantic view and the alignment view respectively, and the search results are fused to obtain the final retrieval result through an adaptive weighted fusion strategy, so as to balance the accuracy of semantic understanding and the real-time response speed of cross-modal association.

[0050] Furthermore, the hierarchical index structure includes: The top-level index uses an inverted index to perform coarse-grained classification by scenario type and task domain, which is used to quickly filter irrelevant data; The underlying index uses product quantization technology to perform fine-grained vector retrieval on the data after coarse-grained classification; The hierarchical index structure supports dynamic updates, enabling the online insertion of newly collected multimodal data and synchronous updates to the relevant index structure. The dynamic update adaptively adjusts the segmentation threshold of the top-level index based on the new sample distribution output by the knowledge update and transfer steps driven by active learning.

[0051] like Figure 6 As shown, the specific steps are as follows: S501. Construct a hierarchical index structure. The top layer uses an inverted index for coarse-grained classification by scenario type and task domain, while the bottom layer uses product quantization to achieve fine-grained vector retrieval. The index structure supports dynamic updates and allows for online insertion and deletion of data entries.

[0052] S502. Design a semantic-aligned dual-view joint retrieval strategy, which performs approximate nearest neighbor search in the semantic view and the alignment view respectively, and then obtains the final retrieval result through adaptive weighted fusion; the fusion weight is dynamically adjusted according to the query type.

[0053] The adaptive weighted fusion strategy is as follows: for queries centered on "object attributes" (such as color and shape), the fusion weight of the semantic view score is increased to 0.7, and the weight of the alignment view score is decreased to 0.3; for queries centered on "task execution" (such as crawling and assembly), the fusion weight of the alignment view score is increased to 0.7, and the weight of the semantic view score is decreased to 0.3, thus balancing the accuracy of semantic understanding with the real-time response speed of cross-modal association.

[0054] S503. Introducing a query rewriting and expansion module, which automatically generates multiple related queries based on the semantic analysis results of the original query, expanding the retrieval coverage; and using semantic clustering methods to extract potentially related concepts from the knowledge base to generate expanded query vectors.

[0055] Specifically, semantic analysis is performed on the original query before retrieval to generate multiple related query vectors; Potentially related concepts are extracted from the knowledge base using semantic clustering methods to generate extended query vectors; The search results of the extended query vector are fused with the search results of the original query vector using a confidence-weighted method to expand the search coverage and improve the search accuracy of complex commands.

[0056] S504. Implement a real-time incremental update mechanism. After the newly collected multimodal data is processed by steps S1-S4, it is written to the database in an asynchronous batch manner, and the relevant index structure is updated at the same time to ensure data freshness.

[0057] The dynamic update of the hierarchical index structure specifically involves: receiving feature distribution statistics (such as mean and variance) from high-uncertainty samples in step S601, and adaptively adjusting the segmentation threshold of the top-level inverted index based on the distribution density of the new samples. For example, when the sample size of a new scenario (such as precision assembly) surges, the number of inverted buckets corresponding to that scenario is automatically increased to achieve adaptive expansion of the index structure and avoid a decrease in retrieval efficiency.

[0058] Step S6: Active learning-driven knowledge updating and transfer Specifically, a sample selection strategy driven by uncertainty is used to screen high-uncertainty samples for expert annotation. The annotation results are then combined with a cross-scenario knowledge transfer framework to update model parameters, achieving dynamic adaptation of the source and target domain distributions and incrementally updating the model without forgetting previous knowledge.

[0059] Furthermore, the uncertainty-driven sample selection strategy specifically includes: Monte Carlo Dropout is used to perform multiple random forward propagations of the model during the inference phase to estimate the variance of the model's predictions. Select high-uncertainty samples with variance greater than a preset threshold and mark them as samples to be labeled; The feature distribution statistics of the high-uncertainty samples are fed back to the cross-scenario knowledge transfer framework to correct the source domain distribution parameters.

[0060] Furthermore, the cross-scenario knowledge transfer framework includes: An adversarial domain adaptation module is constructed to minimize the difference between the source domain data distribution and the target domain data distribution through a game between the generator and the discriminator. A continuous learning mechanism is introduced, and a flexible weight solidification method is used to apply regularization constraints to the key parameters in the model; The regularization intensity of the elastic weight solidification method is dynamically adjusted according to the time alignment confidence. Specifically, when the time alignment confidence is low, the regularization constraints on key parameters are increased to protect learned knowledge; when the time alignment confidence is high, the regularization constraints are decreased to accelerate the model's adaptation to new scenarios.

[0061] The specific steps are as follows: S601. Design an uncertainty-driven sample selection strategy. Predict uncertainty using a Monte Carlo Dropout estimation model, select samples with high uncertainty for expert annotation, and improve annotation efficiency.

[0062] S602. Construct a cross-scenario knowledge transfer framework, reduce the distribution differences between the source domain and the target domain through adversarial domain adaptation methods, and realize the effective transfer of knowledge between different environments.

[0063] S603. Introduce a continuous learning mechanism to incrementally update model parameters without forgetting previous knowledge; adopt an elastic weight solidification method to apply greater regularization constraints to important parameters and protect key knowledge.

[0064] The regularization strength λ of the Elastic Weight Solidification (EWC) is not a fixed constant, but a dynamic function of the time alignment confidence f: λ = λ max ×(1-f), where λ max The maximum regularization strength is preset (e.g., 1000). When f < 0.8 (low confidence, unreliable data), λ is increased to lock the key parameters of the model and prevent catastrophic forgetting based on erroneous data; when f ≥ 0.8 (high confidence, reliable data), λ is decreased to allow the model to quickly learn new scenarios.

[0065] The following detailed implementation of the multimodal embodied intelligent data retrieval method based on dynamic spatiotemporal alignment and view decoupling of the present invention will be provided through specific application scenario examples.

[0066] Example 1: Service robot material handling scenario This embodiment is applied to a service robot material handling scenario, aiming to solve the problem of instruction execution misalignment caused by the spatiotemporal heterogeneity of multimodal data (such as voice latency and real-time visual streams). In this scenario, the robot needs to retrieve a specified item from a shelf based on the user's voice command.

[0067] The multimodal data involved in this scenario includes RGB-D visual data, voice commands, haptic feedback, and joint torque data.

[0068] Step S1: Acquire RGB-D images of the scene using the Kinect depth camera, acquire user voice commands using the microphone array, acquire gripping force feedback using the finger tactile sensor, and acquire robotic arm posture data using the joint encoder.

[0069] Step S2: The time alignment module synchronizes the voice commands (including but not limited to a delay of approximately 200ms) with the visual data (in real time) to handle the end-to-end delay problem of the voice recognition results; the spatial alignment module performs calibration transformation between the camera coordinate system and the robot base coordinate system.

[0070] Step S3: The semantic view encoder extracts the category, position, and shape features of objects in the scene; the alignment view encoder establishes the semantic association between the voice command "Please give me the red box" and the corresponding objects in the visual scene.

[0071] Step S4: The sparse coding module activates relevant experts for "indoor services" and "grabbing operations" based on the current scene type, generates compressed skill representation vectors, and stores them in the local FAISS database.

[0072] Step S5: When the user issues a voice command, the system extracts the semantic vector of the command, retrieves the most matching skill item from the database, and returns the handling strategy for the robot to execute.

[0073] Example 2: Industrial robot assembly scenario This embodiment is applied to an industrial robot assembly scenario to verify the retrieval accuracy of the present invention under strong semantic association requirements. In this scenario, the robot needs to complete the assembly task of parts according to the instructions in the process document.

[0074] The multimodal data in this scenario includes CAD models, engineering drawings, visual guidance, and force control signals.

[0075] Step S1: The CAD model is converted into a point cloud representation and spatially aligned with the RGB-D image of the workpiece collected on site; key parameters of the process document are extracted using OCR and NLP technologies and time-sequentially aligned with the assembly step sequence.

[0076] Step S2: The dynamic spatiotemporal module dynamically updates the timing context according to the assembly progress, expressing the completed steps and the steps to be executed on a unified timeline.

[0077] Step S3: The semantic view encoder extracts the geometric features and surface defect features of the workpiece; the alignment view encoder establishes the matching relationship between process requirements and actual observations.

[0078] Step S4: The sparse coding module activates the "precision assembly" and "bolt tightening" experts according to the assembly stage to generate assembly skill representations.

[0079] Step S5: When the force control signal is abnormal, the system quickly retrieves historical processing experience of similar scenarios to assist in decision-making and adjust the assembly strategy.

[0080] like Figure 7 As shown, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method. Specifically, around the computer storage medium, the present invention constructs six core functional modules, realizing an adaptive closed-loop system of "acquisition-processing-storage-update" through a bidirectional data link: Multimodal data acquisition and spatiotemporal synchronization module: responsible for real-time acquisition of heterogeneous multimodal time-series data, and outputting time alignment confidence through an adaptive time window alignment algorithm; Dynamic spatiotemporal awareness cross-modal representation learning module: Features are extracted using a dual-path Transformer architecture and adaptively fused based on confidence through a gating mechanism; Hierarchical semantic-aligned view decoupling module: Separates semantic view and aligned view using orthogonal constraints to form a joint representation of the two views; Scene-adaptive sparse representation and quantization compression module: Construct a hybrid expert framework and dynamically activate expert subsets to achieve high compression ratio representation; Efficient storage and retrieval module for distributed vector databases: Constructs a hierarchical index structure to support joint retrieval of semantic and aligned dual views; Active learning-driven knowledge update and transfer module: Incremental updates are achieved by screening samples through uncertainty and solidifying them with elastic weights.

[0081] Each module interacts with the central computer storage medium through the aforementioned links, together forming a complete data retrieval and processing system.

[0082] The technical principle of this invention is as follows: Addressing the spatiotemporal heterogeneity, semantic gap, storage efficiency, and environmental adaptability issues of multimodal data in embodied intelligence scenarios, this invention constructs an adaptive closed-loop system of "collection-alignment-representation-decoupling-compression-retrieval-update" using time alignment confidence as the dynamic collaborative thread. Spatiotemporal synchronization layer: Through an adaptive time window alignment algorithm, spatiotemporal synchronization is performed based on sensor sampling rate fluctuations and task urgency, and the output time alignment confidence metric is used to improve quality, thus solving the problem of general alignment ignoring dynamic characteristics.

[0083] Representation learning layer: Features are extracted using a dual-path Transformer (Temporal Attention and Spatial Cross-View Attention), and a gating mechanism is used to adaptively fuse features based on confidence (low confidence focuses on spatial feature compensation, and high confidence focuses on temporal feature capture) to generate joint representations.

[0084] View decoupling layer: Through orthogonal constraints and diversity constraints, it separates fine-grained semantics within a modality (semantic view) from cross-modal task associations (aligned view), bridging the semantic gap.

[0085] Sparse coding layer: Construct a hybrid expert (MoE) framework, dynamically activate expert subsets by combining scene type and confidence, and achieve high compression ratio representation through vector quantization.

[0086] Retrieval layer: It utilizes hierarchical indexes (top-level inverted index, bottom-level product quantization) to support joint retrieval of semantic and aligned dual views, and adaptively merges results according to query type, balancing accuracy and real-time performance.

[0087] Update layer: High uncertainty samples are filtered out by Monte Carlo Dropout, and incremental transfer without forgetting is achieved by combining adversarial domain adaptation and elastic weight solidification (EWC regularization strength is dynamically adjusted with confidence).

[0088] Each module works together dynamically through confidence levels to provide highly reliable data support for the real-time perception-decision closed loop of embodied intelligence.

[0089] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and additions without departing from the principle of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal embodied intelligent data retrieval method based on dynamic spatiotemporal alignment and view decoupling, characterized in that, The method includes: The multimodal data acquisition and spatiotemporal synchronization steps involve real-time acquisition of heterogeneous multimodal time-series data of embodied intelligent devices operating in the physical environment. Through an adaptive time window alignment algorithm, data with different sampling frequencies are mapped to a unified time reference coordinate system based on the dynamic fluctuations of the sensor's real-time sampling rate and the urgency of the task. The quantified time alignment confidence is then output to quantify the alignment quality. The dynamic spatiotemporal awareness cross-modal representation learning step uses a dual-channel Transformer encoder to process the temporal and spatial features of the heterogeneous multimodal time series data respectively. The temporal and spatial features are adaptively weighted and fused through a gating mechanism to generate a joint representation containing spatiotemporal context-aware features. The hierarchical semantic-aligned view decoupling step involves using a semantic view encoder to extract fine-grained semantic information within the modality from the joint representation to construct a semantic view, and using an alignment view encoder to extract cross-modal correlations between task instructions and skill data to construct an alignment view. Orthogonal constraints and diversity constraints are used to ensure that the feature spaces of the semantic view and the alignment view are independent of each other, forming a dual-view joint representation. The scene-adaptive sparse representation and quantization compression steps involve constructing a hybrid expert sparse coding framework. Based on the scene type, task semantic features, and the time alignment confidence, the relevant expert subset in the hybrid expert sparse coding framework is dynamically activated to generate sparse feature representations and perform quantization compression. Efficient storage and retrieval steps for distributed vector databases; constructing a distributed vector database with a hierarchical index structure; supporting semantic-aligned dual-view joint retrieval. The knowledge update and transfer steps driven by active learning use an uncertainty-driven sample selection strategy to screen high-uncertainty samples for expert annotation. The annotation results are then combined with a cross-scenario knowledge transfer framework to update model parameters, achieving dynamic adaptation of the source and target domain distributions and incremental model updates without forgetting previous knowledge.

2. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, The quantized time alignment confidence level is determined in the following way: In the adaptive time window alignment algorithm, cubic spline interpolation is used for low-frequency modes and uniform downsampling is used for high-frequency modes. Cross-modal time correlation is established through the time alignment factor matrix. Calculate the time alignment residual. When the time alignment residual exceeds a preset threshold, mark the time alignment confidence as low and trigger a fault tolerance mechanism. This fault tolerance mechanism adjusts the weight allocation strategy of subsequent feature extraction modules to compensate for time-series deviations. Wherein, the time alignment confidence of the quantization is f∈[0,1].

3. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, In the dual-channel Transformer encoder, the temporal encoder uses the Temporal Attention mechanism to model temporal dependencies in order to capture the dynamic evolution characteristics of multimodal data in the time dimension. The spatial encoder uses the Cross-View Attention mechanism to model cross-view spatial relationships in order to capture the geometric structure and spatial layout features of multimodal data in the spatial dimension; The gating mechanism dynamically adjusts the fusion weights based on the time alignment confidence level, including: increasing the weight ratio of spatial dimension features when the time alignment confidence level is low; When the confidence level of time alignment is high, the weight ratio of time dimension features is increased.

4. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, In the dynamic spatiotemporal perception cross-modal representation learning step, cross-modal alignment is performed at the semantic level, instance level and pixel level through a multi-granularity semantic alignment mechanism, using contrastive learning loss function, triplet ranking loss function and dense matching loss function respectively; Through the multi-granularity semantic alignment mechanism, cross-modal semantic gradient associations from coarse-grained to fine-grained are formed in the feature space.

5. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, The hybrid expert sparse coding framework comprises 64 expert networks, each expert responsible for feature representation in different semantic domains, including motion scenes, natural scenes, man-made object scenes, and interactive scenes. The scene-adaptive sparse representation and quantization compression steps specifically include: Construct a scene classifier and a task encoder to output scene type distribution and task semantic features in real time; The scene type distribution, the task semantic features, and the time alignment confidence are fused in multiple dimensions to generate expert activation weights. Feature calculation is performed only on the expert subset whose weight exceeds a preset threshold. For scenarios with low time alignment confidence, the expert subset with noise resistance and fault tolerance is activated first to generate sparse feature representations.

6. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, In the hierarchical index structure, the top layer uses an inverted index for coarse-grained classification by scene type and task domain, while the bottom layer uses product quantization to achieve fine-grained vector retrieval. The hierarchical index structure supports dynamic updates, enabling the online insertion of newly collected multimodal data and synchronous updates to the relevant index structure. The dynamic update adaptively adjusts the segmentation threshold of the top-level index based on the new sample distribution output by the knowledge update and transfer steps driven by active learning.

7. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, The semantic-aligned dual-view joint retrieval performs approximate nearest neighbor searches in both the semantic view and the alignment view, and obtains the final retrieval result through adaptive weighted fusion. The fusion weight is dynamically adjusted according to the query type.

8. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, The uncertainty-driven sample selection strategy predicts uncertainty using a Monte Carlo Dropout estimation model and selects samples with high uncertainty for expert annotation.

9. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, The cross-scenario knowledge transfer framework includes: The distribution difference between the source and target domains is reduced by using an adversarial domain adaptation method; a continuous learning mechanism is introduced, and a flexible weight solidification method is used to apply regularization constraints to the key parameters in the model. The regularization intensity of the elastic weight solidification method is dynamically adjusted according to the time alignment confidence. Specifically, when the time alignment confidence is low, the regularization constraints on key parameters are increased to protect learned knowledge; when the time alignment confidence is high, the regularization constraints are decreased to accelerate the model's adaptation to new scenarios.

10. The multimodal embodied intelligent data retrieval method according to claim 1, characterized in that, The method further includes: The query rewriting and expansion steps involve semantic analysis of the original query before retrieval to generate multiple related query vectors. Potentially related concepts are extracted from the knowledge base using semantic clustering methods to generate extended query vectors; The search results of the extended query vector are fused with the search results of the original query vector using a confidence-weighted method to expand the search coverage and improve the search accuracy of complex commands.

Images