Retrieval Methods and Systems for Incomplete IoT Cross-Modal Data
By collecting and extracting features through a processor, constructing soft semantic conditions, and learning potential stream transmission, the semantic drift problem caused by modality loss in the Internet of Things is solved, and deterministic recovery and semantic consistency of cross-modal retrieval are achieved.
Patent Information
- Application Number
- CN202611441507.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-17
- Publication Date
- 2026-11-17
AI Technical Summary
Existing cross-modal retrieval methods suffer from semantic drift and loss of instance discrimination information in IoT deployments due to modality missing issues, especially when the proportion of complete paired samples is low, estimation errors are prone to accumulate.
The processor collects multimodal IoT data, extracts modality-specific latent features, constructs prototype soft semantic conditions, learns clean target latent stream transmission, performs reliability-aware gating cross-modal alignment, builds an index, and performs cross-modal retrieval.
It achieves deterministic recovery of the true latent representation of the observed modality from the missing modality without querying the true label, while maintaining semantic consistency and retrieval discriminability.
Smart Images

Figure CN123115487A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of IoT data intelligent processing and multimedia information retrieval technology, and particularly relates to a retrieval method and system for incomplete IoT cross-modal data. Background Technology
[0002] With the development of smart parks, industrial internet, urban governance, and environmental monitoring systems, the same IoT event is often described by multiple modalities, including camera images, device sensor timing data, microphone audio, maintenance text logs, and structured alarm records. Cross-modal retrieval allows querying in one modality to retrieve semantically relevant data from another modality. For example, it can retrieve on-site images based on fault descriptions, maintenance records based on abnormal vibration curves, or corresponding sensor alarms based on inspection images.
[0003] Existing cross-modal retrieval methods typically map different modalities to a shared representation space and bridge the modality gap through relevance learning, metric learning, or deep neural networks. However, existing methods often assume that each training instance has complete and one-to-one correspondences among its modalities. In practical IoT deployments, due to factors such as transmission interruptions, sensor failures, privacy policies, differences in sampling frequencies, differences in storage cycles, and the cost of manual annotation, some modalities of the same event are often missing, resulting in training data containing both complete paired samples and single-modal samples.
[0004] To address the problem of missing modalities, existing solutions obtain missing representations through estimating latent distributions, class-level prototype transfer, neighborhood propagation, or graph-related transfer. However, these solutions typically use estimated distributions, class-level prototypes, neighborhood statistics, or aggregated context as alternative supervisory targets for missing modalities, rather than aiming to recover the true representation of the missing modalities. When the proportion of complete paired samples is low, estimation errors from alternative targets can easily accumulate, leading to semantic drift, overly smoothed recovered features, and loss of instance discrimination information.
[0005] Therefore, a new retrieval method and system for incomplete cross-modal data in the Internet of Things is needed.
[0006] It should be noted that the information disclosed in this background section is only for understanding the background technology of the present application concept, and therefore, the above description is not considered to constitute prior art information. Summary of the Invention
[0007] This disclosure provides at least one method and system for retrieving incomplete cross-modal data from the Internet of Things.
[0008] In a first aspect, embodiments of this disclosure provide a retrieval method for incomplete cross-modal data in the Internet of Things (IoT), including: The processor collects and identifies multimodal IoT data; The processor extracts modality-specific latent features; The processor constructs prototype soft semantic conditions; The processor learns clean target potential stream transmission; Processor reliability-aware gating for cross-modal alignment; The processor builds the index and performs cross-modal retrieval.
[0009] In one optional implementation, the method for the processor to collect and identify multimodal IoT data includes: The processor acquires training samples consisting of at least two IoT data modalities, configures a modality existence mask for each training sample, and divides the training samples into complete paired samples and incomplete samples with at least one missing modality. The modalities of the IoT data include: images, videos, sensor time series, audio, text logs, and structured event records; The modality existence mask is used to indicate whether each modality of each training sample is observable. The reconstruction loss is calculated only for observable modalities, and the recovery of latent features is not used as the reconstruction target of the corresponding modality.
[0010] In one alternative implementation, the method by which the processor extracts mode-specific latent features includes: The processor inputs the observed modal data into the corresponding modality-specific encoder to obtain latent features with the same dimension; the input features are reconstructed by the corresponding decoder for complete paired samples, and the target modality latent features of the complete paired samples are used as the clean target. The latent features and reconstructed features of the m-th mode are respectively and The encoder parameters specific to each mode are not shared; The reconstruction loss is calculated using scaled cosine error: ; Where q≥1.
[0011] In one alternative implementation, the method by which the processor constructs prototype soft semantic conditions includes: The processor constructs semantic prototypes based on the candidate event category names and generates prototype soft semantic conditions based on the similarity between the observed modal latent features and each semantic prototype. Using a frozen pre-trained text encoder to encode the names of C candidate event categories, construct a semantic prototype matrix A: ; Project the latent features of the m-th modality onto the semantic prototype space and calculate the soft probability of the category. ; in, Let τ be the cosine similarity between the projected latent features and the j-th semantic prototype; τ is a parameter. Soft semantic conditions are generated as follows: .
[0012] In one alternative implementation, the method by which the processor learns potential stream transports for clean targets includes: For at least one source mode to target mode direction, the processor establishes a conditional path between the latent features of the source mode in the complete paired samples and the clean latent features of the target mode, and performs conditional path analysis on the latent features of the source mode. and clean latent features of the target mode Construct a linear conditional path on t∈[0,1]. The target speed is Velocity field network with Let be the input, where γ(t) is the temporal embedding and c is the semantic condition, and minimize the squared error between the predicted velocity and the target velocity on complete paired samples, i.e., the latent flow matching loss: ; Among them, E {t,x,y} This expresses the expectation of t, x, y. For incomplete samples with missing modes, the latent features of the missing modes are recovered using Euler numerical integration: ; Where N is the number of calculation steps; k represents the k-th step; The trained neural network stream; The final state serves as a potential feature for recovering the missing target mode.
[0013] In one alternative implementation, the processor reliability-aware gated cross-modal alignment method includes: The processor combines the observed latent features of each training sample with the actual corresponding latent features or recovered latent features to form modality-complete feature pairs, estimates the sample-level modality reliability weights, and performs gated cross-attention fusion to obtain a unified retrieval representation; The data from the two modalities are concatenated to obtain... The scaling cosine loss function is used to measure the alignment error, i.e., the cross-modal consistency loss: ; Where B is the batch size; splicing data Inputting a two-layer fully connected network yields the modal importance weight parameters w: ; Obtaining initial fusion features ; Will As a query, the constructed cross-modal attention fusion is as follows: ; in, Attn(Q;K,V) represents cross-attention fusion features; The final fusion feature is represented as: ; The cross-entropy loss function is, i.e., the semantic classification loss is: ; Where CE represents the cross-entropy loss function, and y represents the true class information.
[0014] In one alternative implementation, the method by which the processor builds an index and performs cross-modal retrieval includes: The processor jointly optimizes the modality reconstruction loss, latent flow matching loss, cross-modal consistency loss, and semantic classification loss, builds a cross-modal index based on the trained unified retrieval representation, and outputs retrieval results based on the similarity between the query representation and the candidate representation.
[0015] Secondly, embodiments of this disclosure also provide a retrieval system employing the above-described retrieval method for incomplete cross-modal IoT data, comprising: The data module is configured to collect and identify multimodal IoT data; An extraction module, configured to extract modality-specific latent features; A constructor module, configured to construct prototype soft semantic conditions; The learning module is configured to learn clean target potential stream transmissions; The alignment module is configured for reliability-aware gated cross-modal alignment. The retrieval module is configured to build indexes and perform cross-modal searches.
[0016] Thirdly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the above-described retrieval method for incomplete cross-modal data of the Internet of Things.
[0017] Fourthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described retrieval method for incomplete cross-modal data of the Internet of Things.
[0018] The beneficial effects of this invention are that the retrieval method for incomplete IoT cross-modal data uses a processor to collect and identify multimodal IoT data; the processor extracts modality-specific latent features; the processor constructs prototype soft semantic conditions; the processor learns clean target latent stream transmission; the processor performs reliability-aware gating of cross-modal alignment; and the processor builds an index and performs cross-modal retrieval. This achieves a deterministic recovery process from observed modalities to the true latent representations of missing modalities using limited complete paired samples, and maintains semantic consistency and retrieval discriminability even without querying true labels.
[0019] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.
[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description
[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0022] Figure 1 A flowchart illustrating a method for retrieving incomplete cross-modal data in the Internet of Things (IoT) provided in this disclosure. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0024] As used herein, the phrases “in one embodiment,” “according to one embodiment,” “in some embodiments,” etc., generally refer to the fact that a particular feature, structure, or characteristic following the phrase can be included in at least one embodiment of this disclosure. Therefore, a particular feature, structure, or characteristic can be included in more than one embodiment of this disclosure, such that these phrases do not necessarily refer to the same embodiment. As used herein, the terms “example,” “exemplary,” etc., are used to “serve as an example, instance, or illustration.” Any implementation, aspect, or design described herein as “example” or “exemplary” is not necessarily to be construed as preferred or superior to other implementations, aspects, or designs. Rather, the use of the terms “example,” “exemplary,” etc., is intended to present concepts in a specific manner.
[0025] To address the problem of missing modalities, existing solutions obtain missing representations through estimating latent distributions, class-level prototype transfer, neighborhood propagation, or graph-related transfer. However, these solutions typically use estimated distributions, class-level prototypes, neighborhood statistics, or aggregated context as alternative supervisory targets for missing modalities, rather than aiming to recover the true representation of the missing modalities. When the proportion of complete paired samples is low, estimation errors from alternative targets can easily accumulate, leading to semantic drift, overly smoothed recovered features, and loss of instance discrimination information.
[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0027] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0028] At least one disclosed embodiment provides a retrieval method for incomplete cross-modal IoT data, comprising: a processor collecting and identifying multimodal IoT data; a processor extracting modality-specific latent features; a processor constructing prototype soft semantic conditions; a processor learning clean target latent stream transmissions; a processor performing reliability-aware gating cross-modal alignment; and a processor building an index and performing cross-modal retrieval. This achieves a deterministic recovery process from observed modalities to the true latent representations of missing modalities using finite complete paired samples, and maintains semantic consistency and retrieval discriminability even without querying true labels.
[0029] In one optional implementation, the method for the processor to collect and identify multimodal IoT data includes: the processor acquiring training samples composed of at least two IoT data modalities, configuring a modality existence mask for each training sample, and dividing the training samples into complete paired samples and incomplete samples with at least one missing modality; the modalities of the IoT data include: images, videos, sensor time series, audio, text logs, and structured event records; the modality existence mask is used to indicate whether each modality of each training sample is observable, the reconstruction loss is calculated only for observable modalities, and the recovery of latent features is not used as the reconstruction target of the corresponding modality.
[0030] In this embodiment, incomplete multimodal IoT data is collected and identified. A modality presence mask is recorded for each event sample, and the data is divided into complete paired samples and incomplete samples with at least one missing modality.
[0031] In this embodiment, the goal is to directly recover the true feature representation of the missing modality, avoiding complete reliance on the estimated distribution, category prototype, or neighborhood average representation as the target for the missing modality. This can alleviate semantic drift and oversmoothing of instance features under high missing rates.
[0032] In one optional implementation, the method for the processor to extract mode-specific latent features includes: inputting each observed mode data into the corresponding mode-specific encoder through the processor to obtain latent features with the same dimension; reconstructing the input features of the complete paired samples through the corresponding decoder, and using the target mode latent features of the complete paired samples as the clean target; The latent features and reconstructed features of the m-th mode are respectively and The encoder parameters specific to each mode are not shared; The reconstruction loss is calculated using scaled cosine error: ; Where q≥1.
[0033] In one optional implementation, the method for the processor to construct prototype soft semantic conditions includes: the processor constructing semantic prototypes based on candidate event category names; generating prototype soft semantic conditions based on the similarity between observed modal latent features and each semantic prototype; and encoding C candidate event category names using a frozen pre-trained text encoder to construct a semantic prototype matrix A. ; Methods for projecting the latent features of the m-th modality onto the semantic prototype space and calculating the soft probability of the category include: ; in, Let τ be the cosine similarity between the projected latent features and the j-th semantic prototype; τ is a parameter. The method for generating soft semantic conditions is as follows: .
[0034] In this embodiment, prototype soft semantic conditions are constructed using candidate event category names. During the inference phase, it is not required to query real labels and can support zero-sample retrieval when the training category and the retrieval category do not overlap.
[0035] In one optional implementation, the method for the processor to learn clean target latent stream transport includes: the processor establishing a conditional path between the source modality latent features of complete paired samples and the clean latent features of the target modality for at least one source modality to target modality direction, and processing the source modality latent features... and clean latent features of the target mode Construct a linear conditional path on t∈[0,1]. The target speed is Velocity field network with Let be the input, where γ(t) is the temporal embedding and c is the semantic condition, and minimize the squared error between the predicted velocity and the target velocity on complete paired samples, i.e., the latent flow matching loss: ; Among them, E {t,x,y} This expresses the expectation of t, x, y. Where N is the number of calculation steps; k represents the k-th step; For the trained neural network stream, the latent features of the missing modes are recovered from incomplete samples with missing modes using Euler numerical integration: ; The final state is used as the latent feature for the recovery of the missing target mode.
[0036] In this embodiment, deterministic continuous transmission is learned through a direction-specific latent flow matching network. Numerical integration can be used to generate latent features of missing modes from observed modes, and the recovery process is stable and interpretable.
[0037] In this embodiment, velocity field networks with specific configuration directions are configured for the source mode to the target mode and the target mode to the source mode, respectively; the inference stage uses the observed latent features as the initial state and employs the Euler integral.
[0038] In this embodiment, the processor establishes a conditional path between the latent features of the source mode and the clean latent features of the target mode in the direction from at least one source mode to the target mode, and trains a latent flow matching recovery network to predict the velocity field of the conditional path. For incomplete samples, the observed latent features of the mode and the prototype soft semantic conditions are input into the latent flow matching recovery network, and the recovered latent features of the missing target mode are obtained by numerical integration.
[0039] In one optional implementation, the processor reliability-aware gated cross-modal alignment method includes: the processor combining the observed latent features of each training sample with the true corresponding latent features or recovered latent features to form modal complete feature pairs, estimating sample-level modal reliability weights and performing gated cross-attention fusion to obtain a unified retrieval representation while maintaining intra-modal and inter-modal semantic structures; and concatenating the data from the two modalities to obtain... The scaling cosine loss function is used to measure the alignment error, i.e., the cross-modal consistency loss: ; Where B is the batch size; splicing data Inputting a two-layer fully connected network yields the modal importance weight parameters w: ; Obtaining initial fusion features ; Will As a query, the constructed cross-modal attention fusion is as follows: ; in, Attn(Q;K,V) represents cross-attention fusion features; The final fusion feature is represented as: ; The cross-entropy loss function, i.e., the semantic classification loss, is: ; Where CE represents the cross-entropy loss function, and y represents the true class information.
[0040] In this embodiment, the observed or recovered modal features are concatenated and input into a two-layer fully connected network, and the reliability weights of each modality are obtained through softmax. A linear fusion representation is obtained based on the reliability weights of each modality. The linear fusion representation is used as a query to perform multi-head scaling dot product cross attention on each modal feature, and the contribution of the cross attention residual is controlled by a zero-initialized learnable gating parameter.
[0041] In one optional implementation, the method by which the processor builds an index and performs cross-modal retrieval includes: the processor jointly optimizing modality reconstruction loss, latent flow matching loss, cross-modal consistency loss and semantic classification loss, building a cross-modal index based on the trained unified retrieval representation, and outputting retrieval results according to the similarity between the query representation and the candidate representation.
[0042] In this embodiment, by gradually fusing observed features and recovered features through sample-level reliability weights and zero-initialization gated cross-attention, the adverse effects of low-reliability recovery results on the unified retrieval representation can be suppressed.
[0043] In this embodiment, the modality-specific encoder can be a visual Transformer or convolutional network, a temporal Transformer or a one-dimensional convolutional network, an audio spectrogram encoder, or a text encoder, depending on the data type; each encoder has the same output dimension, but the parameters are not shared between them.
[0044] In this embodiment, the reconstruction loss is calculated only for the actual observed modes, and the recovered representation is not used as the reconstruction target, so as to prevent the model from repeatedly amplifying its own generation error as a supervision signal.
[0045] In this embodiment, the flow matching loss is calculated only from complete paired samples; the latent features obtained from the recovery of incomplete samples participate in the cross-modal consistency loss, classification loss, and similarity structure preservation loss, thereby utilizing incomplete samples without contaminating the clean recovery endpoint.
[0046] In this embodiment, real category prototypes can be used as semantic conditions when training the stream matching network; prototype soft semantic conditions are used when recovering incomplete samples and retrieval reasoning, without needing to query real labels.
[0047] In this embodiment, the overall training objective is: ; Wherein, L1 is the reconstruction loss, i.e., modal reconstruction loss; L2 is the latent flow matching loss; L3 is the cross-modal consistency loss; and L4 is the semantic classification loss. The weights for each loss.
[0048] In this embodiment, the joint maintenance of intramodal and intermodal similar structures is applicable to IoT applications such as equipment inspection, fault location, alarm correlation, event tracing, and multi-source data governance.
[0049] At least one other disclosed embodiment also provides a retrieval system employing the above-described retrieval method for incomplete IoT cross-modal data, comprising: a data module configured to collect and identify multimodal IoT data; an extraction module configured to extract modality-specific latent features; a construction module configured to construct prototype soft semantic conditions; a learning module configured to learn clean target latent stream transmissions; an alignment module configured to perform reliability-aware gating cross-modal alignment; and a retrieval module configured to build an index and perform cross-modal retrieval.
[0050] At least one other disclosed embodiment also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described retrieval method for incomplete Internet of Things cross-modal data.
[0051] At least one other disclosed embodiment also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described retrieval method for incomplete IoT cross-modal data.
[0052] In summary, this retrieval method for incomplete IoT cross-modal data achieves a deterministic recovery process from observed modalities to missing modalities using finite complete paired samples. This method maintains semantic consistency and retrieval discriminability even without query labels. It involves: processor collecting and identifying multimodal IoT data; processor extracting modal-specific latent features; processor constructing prototype soft semantic conditions; processor learning clean target latent stream transmission; processor reliability-aware gating of cross-modal alignment; and processor building an index and performing cross-modal retrieval.
[0053] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.
Claims
1. A retrieval method for incomplete cross-modal data in the Internet of Things, characterized in that, include: The processor collects and identifies multimodal IoT data; The processor extracts modality-specific latent features; The processor constructs prototype soft semantic conditions; The processor learns clean target potential stream transmission; Processor reliability-aware gating for cross-modal alignment; The processor builds the index and performs cross-modal retrieval.
2. The retrieval method for incomplete cross-modal data of the Internet of Things as described in claim 1, characterized in that: The method for the processor to collect and identify multimodal IoT data includes: The processor acquires training samples consisting of at least two IoT data modalities, configures a modality existence mask for each training sample, and divides the training samples into complete paired samples and incomplete samples with at least one missing modality. The modalities of the IoT data include: images, videos, sensor time series, audio, text logs, and structured event records; The modality existence mask is used to indicate whether each modality of each training sample is observable. The reconstruction loss is calculated only for observable modalities, and the recovery of latent features is not used as the reconstruction target of the corresponding modality.
3. The retrieval method for incomplete cross-modal data of the Internet of Things as described in claim 1, characterized in that: The method for extracting modality-specific latent features by the processor includes: The processor inputs the observed modal data into the corresponding modality-specific encoder to obtain latent features with the same dimension; the input features are reconstructed by the corresponding decoder for complete paired samples, and the target modality latent features of the complete paired samples are used as the clean target. The latent features and reconstructed features of the m-th modality are respectively and The encoder parameters specific to each mode are not shared; The reconstruction loss is calculated using scaled cosine error: ; Where q≥1.
4. The retrieval method for incomplete cross-modal data of the Internet of Things as described in claim 1, characterized in that: The method for the processor to construct prototype soft semantic conditions includes: The processor constructs semantic prototypes based on the candidate event category names and generates prototype soft semantic conditions based on the similarity between the observed modal latent features and each semantic prototype. Using a frozen pre-trained text encoder to encode the names of C candidate event categories, construct a semantic prototype matrix A: ; Project the latent features of the m-th modality onto the semantic prototype space and calculate the soft probability of the category: ; in, Let τ be the cosine similarity between the projected latent features and the j-th semantic prototype; τ is a parameter. Soft semantic conditions are generated as follows: 。 5. The retrieval method for incomplete cross-modal data of the Internet of Things as described in claim 1, characterized in that: The method for the processor to learn potential stream transports for clean targets includes: The processor establishes a conditional path between the latent features of the source mode and the clean latent features of the target mode in the direction from at least one source mode to the target mode, and performs conditional path analysis on the latent features of the source mode. and clean latent features of the target mode Construct a linear conditional path on t∈[0,1]. The target speed is Velocity field network with Let be the input, where γ(t) is the temporal embedding and c is the semantic condition, and minimize the squared error between the predicted velocity and the target velocity on complete paired samples, i.e., the latent flow matching loss: ; Among them, E {t,x,y} This expresses the expectation of t, x, y. For incomplete samples with missing modes, the latent features of the missing modes are recovered using Euler numerical integration: ; Where N is the number of calculation steps; k represents the k-th step; For a trained neural network stream, the final state serves as a latent feature for recovering the missing target mode.
6. The retrieval method for incomplete cross-modal data of the Internet of Things as described in claim 1, characterized in that: The processor reliability-aware gated cross-modal alignment method includes: The processor combines the observed latent features of each training sample with the actual corresponding latent features or recovered latent features to form modality-complete feature pairs, estimates the sample-level modality reliability weights, and performs gated cross-attention fusion to obtain a unified retrieval representation; The data from the two modalities are concatenated to obtain... The scaling cosine loss function is used to measure the alignment error, i.e., the cross-modal consistency loss: ; Where B is the batch size; splicing data Inputting a two-layer fully connected network yields the modal importance weight parameters w: ; Obtaining initial fusion features ; Will As a query, the constructed cross-modal attention fusion is as follows: ; in, Attn(Q;K,V) represents cross-attention fusion features; The final fusion feature is represented as: ; The cross-entropy loss function, i.e., the semantic classification loss, is: ; Where CE represents the cross-entropy loss function, and y represents the true class information.
7. The retrieval method for incomplete cross-modal data of the Internet of Things as described in claim 1, characterized in that: The method for the processor to build an index and perform cross-modal retrieval includes: The processor jointly optimizes the reconstruction loss, latent flow matching loss, cross-modal consistency loss, and semantic classification loss, builds a cross-modal index based on the trained unified retrieval representation, and outputs retrieval results based on the similarity between the query representation and the candidate representation.
8. A retrieval system employing the retrieval method for incomplete cross-modal data of the Internet of Things as described in any one of claims 1-7, characterized in that, include: The data module is configured to collect and identify multimodal IoT data; An extraction module, configured to extract modality-specific latent features; A constructor module, configured to construct prototype soft semantic conditions; The learning module is configured to learn clean target potential stream transmissions; The alignment module is configured for reliability-aware gated cross-modal alignment. The retrieval module is configured to build indexes and perform cross-modal searches.
9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When executed by a processor, the computer program / instruction implements the steps of the retrieval method for incomplete cross-modal data of the Internet of Things as described in any one of claims 1-7.
10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the steps of the retrieval method for incomplete cross-modal data of the Internet of Things as described in any one of claims 1-7.
