A method and system for recovering incomplete multimodal perception data based on conditional diffusion

Abstract

The present invention provides an incomplete multimodal perception data recovery method and system based on conditional diffusion, the method comprising the following steps: sensing the reception status of data of multiple modalities in real time, determining whether there is data missing in at least one modality; constructing a Gaussian process based on historical data of the modality of missing data, constructing a conditional distribution based on the Gaussian process, sampling from the conditional distribution, and obtaining a time correlation matrix; determining the time period of missing data of the modality of missing data, obtaining data of the corresponding time period from the modality of non-missing data, constructing each feature matrix corresponding to each modality of non-missing data, fusing the feature matrices of multiple modalities, and obtaining a semantic consistency matrix; inputting the data of the conditional distribution, the time correlation matrix, and the semantic consistency matrix into a preset feature recovery model, the feature recovery model outputting a recovery feature matrix of the missing data of the modality corresponding to the missing data.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method and system for recovering incomplete multimodal perception data based on conditional diffusion. Background Art

[0002] With the rapid development of IoT technology and the continuous improvement of computing power, the types and number of sensor devices have grown exponentially, giving rise to massive amounts of multimodal data. These data come from a variety of sensor types, including image sensors (cameras), temperature and humidity sensors, light sensors, sound sensors, and various signal detection devices. These sensors can not only record the physical state of the environment, but also capture subtle changes in different dimensions, providing key data support for intelligent monitoring, anomaly detection, environmental perception, and automatic decision-making. In this context, integrating multimodal data has become the key to fully capture multiple influencing factors in complex scenes, and is of great significance in IoT applications. First, image data can provide visual information of the overall scene, while high-frequency signal data can continuously track tiny dynamic changes. For example, in intelligent environmental monitoring, combining image data captured by mobile phone cameras and signal data from site sensors can help you understand air pollution anytime and anywhere. Second, data of different modalities have different sensitivities to specific factors and can perceive environmental changes in multiple dimensions. For example, in smart factories, by combining 3D point clouds and RGB images, tiny defects in products can be detected, achieving more efficient anomaly detection. Therefore, each modal data can complement each other in monitoring specific factors, making the system have a more comprehensive understanding of the environment. By integrating multimodal data, the IoT system can have a deeper understanding of environmental dynamics and provide rich multi-dimensional information support for intelligent decision-making, thereby improving the system's adaptability, accuracy and intelligence.

[0003] However, IoT devices are usually deployed in environments with complex network conditions and are affected by multiple factors such as network fluctuations, sensor failures, and data delays. This can easily lead to partial data loss in individual modalities at the same time. This is particularly common in large-scale sensor networks, affecting the overall analysis of multiple modalities. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an incomplete multimodal perception data recovery method based on conditional diffusion to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides an incomplete multimodal perception data recovery method based on conditional diffusion, the method comprising the steps of:

[0006] Real-time perception of the reception of data in multiple modes, and determination of whether data in at least one mode is missing;

[0007] constructing a Gaussian process based on historical data of the mode of missing data, constructing a conditional distribution based on the Gaussian process, sampling from the conditional distribution, and obtaining a time correlation matrix;

[0008] Determine the time period of missing data for the modality with missing data, obtain the data of the corresponding time period from the modality with non-missing data, construct a feature matrix corresponding to each modality with non-missing data, fuse the feature matrices of multiple modalities, and obtain a semantic consistency matrix;

[0009] The conditionally distributed data, the time correlation matrix and the semantic consistency matrix are input into a preset feature recovery model, and the feature recovery model outputs a recovery feature matrix of the missing data corresponding to the modality of the missing data.

[0010] The above scheme is adopted. In this scheme, the data of the modalities are received in real time. When it is determined that there is a missing data, on the one hand, a time correlation matrix is ​​constructed based on the time correlation of the modality through the historical data of the modality with missing data. On the other hand, a semantic consistency matrix is ​​constructed based on the semantic consistency between the modalities through the data of the modalities that are not missing in the same time period. The time correlation matrix and the semantic consistency matrix are used as input data, and data recovery is performed through information from multiple aspects to ensure data recovery accuracy.

[0011] In some embodiments of the present invention, in the step of inputting conditionally distributed data, temporal correlation matrix and semantic consistency matrix into a preset feature recovery model, wherein the feature recovery model outputs a recovery feature matrix of missing data corresponding to the modality of the missing data, the feature recovery model is a diffusion model.

[0012] In some embodiments of the present invention, in the step of inputting conditionally distributed data, temporal correlation matrix and semantic consistency matrix into a preset feature recovery model, wherein the feature recovery model outputs a recovered feature matrix of missing data corresponding to the modality of the missing data, the feature recovery model is a noise-based fractional model.

[0013] In some embodiments of the present invention, if the feature recovery model is a noise-based score model, then in the step of inputting the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix into a preset feature recovery model, and the feature recovery model outputting a recovery feature matrix of missing data corresponding to the modality of the missing data:

[0014] Constructing a fusion matrix based on the temporal correlation matrix and the semantic consistency matrix;

[0015] The feature recovery model outputs a trajectory matrix based on the fusion matrix;

[0016] A restored feature matrix is ​​obtained based on the trajectory matrix and the fusion matrix.

[0017] In some embodiments of the present invention, in the step of obtaining a restored feature matrix based on the trajectory matrix and the fusion matrix, the restored feature matrix is ​​calculated based on the following formula:

[0018] = +

[0019] in, represents the restored feature matrix, represents the fusion matrix, represents the trajectory matrix.

[0020] In some embodiments of the present invention, in the step of obtaining data for a corresponding time period from a modality with no missing data and constructing a feature matrix corresponding to each modality with no missing data:

[0021] Obtain the data for the corresponding time period from the mode with no missing data, and construct an initial matrix based on the data at each time point in the time period;

[0022] The initial matrix is ​​convolved to obtain a feature matrix corresponding to each mode.

[0023] In some embodiments of the present invention, in the step of constructing an initial matrix based on data at each time point in the time period, the data at each time point in the time period is encoded, and the encoded data at each time point are combined to construct the initial matrix.

[0024] In some embodiments of the present invention, in the step of fusing feature matrices of multiple modalities to obtain a semantic consistency matrix, the feature matrices of multiple modalities are convolved to obtain the semantic consistency matrix.

[0025] In some embodiments of the present invention, the method further comprises training optimization, and training is performed based on the following first loss function and second loss function:

[0026]

[0027] in, represents the value of the first loss function, represents the fusion matrix, The label feature matrix representing missing data, represents the trajectory matrix;

[0028]

[0029] in, represents the restored feature matrix, represents the value of the second loss function, Label feature matrix representing missing data.

[0030] The second aspect of the present invention also provides an incomplete multimodal perception data recovery system based on conditional diffusion, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0031] The third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned incomplete multimodal perception data recovery method based on conditional diffusion.

[0032] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.

[0033] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0035] Figure 1 A schematic diagram of an implementation of an incomplete multimodal perception data recovery method based on conditional diffusion of the present invention;

[0036] Figure 2 It is a schematic diagram of the architecture of the incomplete multimodal perception data recovery method based on conditional diffusion of the present invention;

[0037] Figure 3 Schematic diagram of the diffusion process of the present invention and the traditional diffusion process. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0039] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0040] In the specific implementation process, there are significant differences in the layout location, data collection frequency and data type of various sensors in the IoT system, which leads to the high heterogeneity of data sources. First, various sensors are usually scattered in complex network topologies and have inconsistent collection frequencies: for example, image sensors may sample at a frequency of seconds, while temperature and humidity sensors may sample at a period of minutes or longer. This difference in data collection frequency increases the complexity of multimodal data fusion. In addition, IoT devices are usually deployed in environments with complex network conditions, and are affected by multiple factors such as network fluctuations, sensor failures, and data delays, which can easily lead to missing or incomplete data, which is particularly common in large-scale sensor networks. For example, a camera may not be able to obtain image data due to environmental occlusion or hardware failure, while an environmental sensor may not be able to transmit data on time due to insufficient power or network interruption. This lack of multimodal data poses a severe challenge to the stability and analysis accuracy of the IoT system. In the process of data fusion, data missing may cause the system to have a deviation in its understanding of the actual situation, affecting the real-time nature of intelligent monitoring, the accuracy of anomaly detection, and the effectiveness of decision support. For example, in an intelligent transportation system, incomplete sensor data will affect the judgment of road traffic flow and lead to errors in traffic control decisions. Therefore, how to effectively restore missing data when multimodal perception data is incomplete has become a key technical challenge in IoT intelligent analysis. Solving this challenge not only helps improve the robustness and response speed of IoT systems, but also lays the foundation for multimodal data fusion and intelligent decision-making in complex environments.

[0041] Therefore, designing a data recovery method that can adapt to incomplete multimodal data and has high recovery accuracy is the key to multimodal sensing data processing in the Internet of Things. When a sensor data is disturbed or lost, other modal data can provide key information and maintain the continuity of monitoring and analysis.

[0042] Prior art 1

[0043] The specific plan is:

[0044] Multimodal data recovery technology based on graph neural network: for example, GCNet designed specifically for conversational data [3] The network aims to solve the problem of incomplete multimodal data (sound, text, video) in conversations. The specific approach includes: (1) Node construction: Each conversation contains multiple utterances, and each utterance is Represented as a node In order to extract The initial representation of the network is constructed by first connecting incomplete multimodal features, and then using Bi-LSTM to capture contextual information. (2) Speaker and Time GNN: Speaker GNN and Time GNN are key modules for capturing speaker and time dependencies in conversations. In these modules, edges are used to measure the importance of connections between nodes. Edge types define different aggregation methods in node interactions. Speaker GNN and Time GNN share the same edges but different edge types to capture different dependencies. (3) Classification and Reconstruction: For each node, its initial representation, representation considering speaker information, and representation considering time information are extracted. To aggregate these representations, they are connected together and then Bi-LSTM is used for context-sensitive modeling.

[0045] Disadvantages of the prior art 1

[0046] (1) It relies on GNN to model the dependency of multimodal data, but GNN has high computational overhead when processing large-scale data and is difficult to expand to large-scale IoT application environments. (2) The performance degradation is more obvious when processing data with high missing rates, which shows that GNN's ability to recover data is still insufficient when the missing modality is serious. (3) The recovery technology based on GNN mainly focuses on the loss of the entire modality and cannot handle the situation where part of the data is missing within the modality. The perception data in the IoT is often a time series signal collected from sensors. When this type of data is divided into windows for processing, it is more likely to have partial missing within the window rather than complete missing of the entire window. (4) The GCNet network is mainly designed for conversational data, and its model structure and dependent information (such as speaker dependency and time dependency) may not be applicable in other application scenarios.

[0047] Prior art 2

[0048] The specific plan is:

[0049] Multimodal data recovery technology based on translation network: For example, the MCTN network (Multimodal Cyclic Translation Network) designed for sentiment analysis achieves robust representation learning of multimodal data through cyclic translation between modalities. Its specific implementation includes: (1) Modal cyclic translation: First, the source modality data is mapped to the target modality representation through the cyclic translation model, and then reconstructed back to the source modality through the decoder to ensure the integrity of the information transmitted between modalities by the intermediate representation. (2) Encoder-decoder structure: The model uses the Seq2Seq architecture for modality translation. The encoder is responsible for converting the source modality into the intermediate representation, and the decoder is responsible for mapping it to the target modality or reversely restoring it to the source modality. (3) Translation consistency loss: During the training process, the cyclic consistency loss is used to ensure that the representation learned by the model can retain information integrity between multiple modalities. Through this loss mechanism, MCTN can rely only on the source modality data without the target modality during the test phase, and can still maintain prediction performance even if some modalities are missing or interfered with during testing.

[0050] Disadvantages of the second prior art:

[0051] (1) Strong dependence on sequence data: MCTN mainly relies on the Seq2Seq structure for inter-modal translation, which makes it perform well when processing time series data, but has limited applicability in application scenarios of non-sequential data or unstructured data, resulting in limited generalization ability. (2) Complexity of cycle consistency training: Although cycle consistency loss helps to fully transfer information, it will introduce additional optimization difficulties in more modal structures, making the training process complex and easily affected by the weight of the loss term, thus requiring more sophisticated hyperparameter adjustment to achieve the best effect. (3) Limited ability to handle high missing rates: Although MCTN maintains a certain degree of robustness on multimodal data with partial missing data, its performance will significantly decrease in the case of severe missing data (such as most modal data missing or a large number of missing data within a single modal data). (4) Strong dependence on the target task: MCTN is mainly designed for sentiment analysis tasks. Its architecture and loss function optimization objectives are closely centered on sentiment classification problems. Therefore, when applied to multimodal tasks of other IoT perception data, a lot of architectural adjustments or redesign may be required.

[0052] like Figure 1 and 2 As shown, the present invention proposes an incomplete multimodal perception data recovery method based on conditional diffusion, and the steps of the method include:

[0053] Step S100, sensing the reception status of data in multiple modes in real time, and determining whether there is a lack of data in at least one mode;

[0054] In some embodiments of the present invention, in the process of weather forecasting, it is necessary to collect data of multiple modes such as temperature, light and humidity.

[0055] In some embodiments of the present invention, when missing data of any modality is detected, the processing of this solution is started.

[0056] In the specific implementation, considering that each modality has different data structures and unique features, this scheme uses a shallow feature extractor to effectively capture modality-specific features. To coordinate heterogeneous sensory data and promote their integration, this module configures a separate encoder for each modality.

[0057] In the specific implementation process, in order to coordinate heterogeneous perception data ( ) and facilitate their integration, the module configures a separate encoder for each modality, aiming to project different inputs into a dimensional vector representation Therefore, the process of shallow feature extraction can be expressed as:

[0058] ;

[0059] Among them, the encoded features It is divided into two parts accordingly: observable features and missing features . The parameter is The encoder.

[0060] Step S200, constructing a Gaussian process based on historical data of the mode of missing data, constructing a conditional distribution based on the Gaussian process, sampling from the conditional distribution, and obtaining a time correlation matrix;

[0061] In some embodiments of the present invention, in the step based on the conditional distribution of data, the observable part of the intra-modal data is used to construct a Gaussian process, thereby increasing the prior knowledge of the missing data. Specifically, this scheme constructs a matrix of the modal historical data in the latent space and the matrix of missing data The joint distribution of:

[0062]

[0063] Specifically, the mean is set to zero. Using the kernel function as a covariance matrix, capturing and Therefore, the conditional distribution It can be calculated as follows:

[0064]

[0065] Based on this, | can be used as a prior distribution of the latent space of missing data, From the conditional distribution Sampling comes from, and s represents the number of diffusion steps.

[0066] Step S300, determining the time period of missing data of the modality with missing data, obtaining data of the corresponding time period from the modality with non-missing data, constructing a feature matrix corresponding to each modality with non-missing data, and fusing the feature matrices of multiple modalities to obtain a semantic consistency matrix;

[0067] Step S400, inputting the conditionally distributed data, the time correlation matrix and the semantic consistency matrix into a preset feature recovery model, the feature recovery model outputs a recovery feature matrix of the missing data corresponding to the modality of the missing data.

[0068] like Figure 2 As shown, Figure 2 Medium Mode is the mode of missing data, mode and is the mode with no missing data; , and Respectively represent the shallow feature extractors for the three modalities; , and Respectively express , b and Feature data after shallow feature extraction of three modal raw data; N (0, ) represents the conditional distribution of the mode of missing data; N(0, ) and N(0, ) means if and The conditional distribution of the mode with missing data when is the mode with no missing data; , and They represent the noise-based fractional models of the corresponding modal settings, through which the inverse ODE (Ordinary Differential Equation) is completed; P( )、P( ) and P( ) represent the true distribution of each mode (if there is a missing part), represents the corresponding restored feature matrix, Represents the complete mode consisting of the restored characteristic matrix and the initial existing characteristic matrix The characteristic data of and Respectively represent the convolutional networks for the complete modes b and o, and They represent the characteristic matrices of the corresponding conditional constraints of the complete modes b and o respectively.

[0069] The above scheme is adopted. In this scheme, the data of the modalities are received in real time. When it is determined that there is a missing data, on the one hand, a time correlation matrix is ​​constructed based on the time correlation of the modality through the historical data of the modality with missing data. On the other hand, a semantic consistency matrix is ​​constructed based on the semantic consistency between the modalities through the data of the modalities that are not missing in the same time period. The time correlation matrix and the semantic consistency matrix are used as input data, and data recovery is performed through information from multiple aspects to ensure data recovery accuracy.

[0070] In some embodiments of the present invention, in the step of inputting conditionally distributed data, temporal correlation matrix and semantic consistency matrix into a preset feature recovery model, wherein the feature recovery model outputs a recovery feature matrix of missing data corresponding to the modality of the missing data, the feature recovery model is a diffusion model.

[0071] In the specific implementation process, in the step of inputting the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix into a preset feature recovery model, the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix are constructed into an input matrix through a preset conditional enhancer, and the input matrix is ​​input into the preset feature recovery model.

[0072] In a specific implementation process, the conditional enhancer may be a convolutional neural network or a conditional enhanced adversarial network, etc.

[0073] In some embodiments of the present invention, in the step of inputting conditionally distributed data, temporal correlation matrix and semantic consistency matrix into a preset feature recovery model, wherein the feature recovery model outputs a recovered feature matrix of missing data corresponding to the modality of the missing data, the feature recovery model is a noise-based fractional model.

[0074] Using the above scheme, the diffusion process of this scheme is completed through one-step diffusion, which improves processing efficiency. The noise-based score model is a score-based neural network, which is used to approximate the speed function of the conversion, thereby ensuring that the conversion process between the two states of the one-step diffusion of this scheme closely follows a straight line trajectory to achieve one-step recovery. In addition, this scheme adds two key condition constraints in the conversion process: the time correlation within each modality and the semantic consistency between different modalities.

[0075] In some embodiments of the present invention, if the feature recovery model is a noise-based score model, then in the step of inputting the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix into a preset feature recovery model, and the feature recovery model outputting a recovery feature matrix of missing data corresponding to the modality of the missing data:

[0076] Constructing a fusion matrix based on the temporal correlation matrix and the semantic consistency matrix;

[0077] The feature recovery model outputs a trajectory matrix based on the fusion matrix;

[0078] A restored feature matrix is ​​obtained based on the trajectory matrix and the fusion matrix.

[0079] In some embodiments of the present invention, in the step of obtaining a restored feature matrix based on the trajectory matrix and the fusion matrix, the restored feature matrix is ​​calculated based on the following formula:

[0080] = +

[0081] in, represents the restored feature matrix, represents the fusion matrix, represents the trajectory matrix.

[0082] In the specific implementation process, in order to reduce the decision-making errors caused by incomplete multimodal perception data, this scheme uses the ODE diffusion model to generate missing data distribution based on the existing available data, while maintaining the semantic consistency between different modalities and the sequence correlation within the modality, thereby achieving more accurate and reliable multimodal data recovery.

[0083] Using the above scheme, the traditional diffusion process requires a trade-off between computational efficiency and sample quality, because from the flow point of view, it introduces unnecessary curvature to the generated path, such as Figure 3 As shown in (a), is the distribution of the initial state before diffusion, Figure 3 middle is the distribution after multi-step diffusion, is the distribution after one-step diffusion, and the ODE-based diffusion method of this scheme can generate nearly straight-line flow in one step, improve the processing efficiency, and have good performance, such as Figure 3 Therefore, this scheme uses a diffusion method based on ODE (Ordinary Differential Equation) to achieve incomplete multimodal perception data recovery.

[0084] In some embodiments of the present invention, in the step of obtaining data for a corresponding time period from a modality with no missing data and constructing a feature matrix corresponding to each modality with no missing data:

[0085] Obtain the data for the corresponding time period from the mode with no missing data, and construct an initial matrix based on the data at each time point in the time period;

[0086] The initial matrix is ​​convolved to obtain a feature matrix corresponding to each mode.

[0087] In a specific implementation process, the convolution processing is a one-dimensional convolution processing.

[0088] In some embodiments of the present invention, in the step of constructing an initial matrix based on data at each time point in the time period, the data at each time point in the time period is encoded, and the encoded data at each time point are combined to construct the initial matrix.

[0089] In some embodiments of the present invention, the encoder adopts a connection structure of a single-layer LSTM and U-Net, which are used for feature extraction of sequence data and visual data, respectively, and other models can also be used instead. To ensure the robust representation of each modality, this scheme adopts a two-stage training strategy. First, the shallow feature extractor is pre-trained in a complete multimodal mode to effectively identify and encode modality-specific features. At this stage, each feature encoder is optimized by a unimodal loss function, aiming to minimize the reconstruction error, thereby improving the fidelity of each modality in the latent representation space. On this basis, the subsequent modality recovery process focuses on reconstructing the shallow features of these missing modalities based on existing information.

[0090] In some embodiments of the present invention, in the step of fusing feature matrices of multiple modalities to obtain a semantic consistency matrix, the feature matrices of multiple modalities are convolved to obtain the semantic consistency matrix.

[0091] In some embodiments of the present invention, the method further comprises training optimization, and training is performed based on the following first loss function and second loss function:

[0092]

[0093] in, represents the value of the first loss function, represents the fusion matrix, The label feature matrix representing missing data, represents the trajectory matrix;

[0094]

[0095] in, represents the restored feature matrix, represents the value of the second loss function, Label feature matrix representing missing data.

[0096] In the specific implementation process, the entire network of this scheme is trained as a whole through the first loss function and the second loss function. The entire network of this scheme includes a shallow feature extractor, a convolution layer for convolution processing, a conditional enhancer and a feature recovery model.

[0097] In summary, the beneficial effects of this solution include:

[0098] 1. In order to reduce the decision-making errors caused by incomplete multimodal perception data, the network structure of this scheme can achieve more accurate and reliable multimodal data recovery.

[0099] 2. This scheme can generate missing data in one step by establishing a diffusion trajectory between the distribution of available data and missing data.

[0100] 3. In order to enhance the accuracy and reliability of recovery, this scheme adds two conditional constraints: sequential correlation based on Gaussian process and semantic consistency based on learnable content, to enhance the dependency within and between modalities and ensure the accuracy of data recovery.

[0101] An embodiment of the present invention also provides an incomplete multimodal perception data recovery system based on conditional diffusion, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0102] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the aforementioned incomplete multimodal perception data recovery method based on conditional diffusion are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0103] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0104] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0105] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for restoring incomplete multimodal perception data based on conditional diffusion, characterized in that: The steps of the method include: Real-time perception of the reception of data in multiple modes, and determination of whether data in at least one mode is missing, where the data in multiple modes include temperature, light, and humidity data; A Gaussian process is constructed based on the historical data of the modality of missing data, a conditional distribution is constructed based on the Gaussian process, a time correlation matrix is ​​obtained by sampling from the conditional distribution, and a matrix of the modal historical data is constructed in the latent space. and the matrix of missing data The joint distribution of: Mean is set to zero, using the kernel function as a covariance matrix, capturing and The temporal correlation between The calculation is as follows: in, | As the prior distribution of the latent space of missing data, s represents the number of diffusion steps; Determine the time period of missing data for the modality with missing data, obtain data for the corresponding time period from the modality with non-missing data, construct a feature matrix corresponding to each modality with non-missing data, fuse the feature matrices of multiple modalities, and obtain a semantic consistency matrix; The conditionally distributed data, the time correlation matrix and the semantic consistency matrix are input into a preset feature recovery model, and the feature recovery model outputs a recovery feature matrix of the missing data corresponding to the modality of the missing data.

2. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 1 is characterized in that: In the step of inputting the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix into a preset feature recovery model, the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix are constructed into an input matrix through a preset conditional enhancer, and the input matrix is ​​input into the preset feature recovery model, and the conditional enhancer is a convolutional neural network or a conditional enhanced adversarial network.

3. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 1 is characterized in that: In the step of inputting conditionally distributed data, temporal correlation matrix and semantic consistency matrix into a preset feature recovery model, wherein the feature recovery model outputs a restored feature matrix of missing data corresponding to the modality of the missing data, the feature recovery model is a noise-based score model.

4. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 3 is characterized in that: If the feature recovery model is a noise-based score model, in the step of inputting the conditionally distributed data, the temporal correlation matrix and the semantic consistency matrix into a preset feature recovery model, and the feature recovery model outputting a recovery feature matrix of missing data corresponding to the modality of the missing data: Constructing a fusion matrix based on the temporal correlation matrix and the semantic consistency matrix; The feature recovery model outputs a trajectory matrix based on the fusion matrix; A restored feature matrix is ​​obtained based on the trajectory matrix and the fusion matrix.

5. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 4 is characterized in that: In the step of obtaining a restored feature matrix based on the trajectory matrix and the fusion matrix, the restored feature matrix is ​​calculated based on the following formula: = + in, represents the restored feature matrix, represents the fusion matrix, represents the trajectory matrix.

6. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 1 is characterized in that: In the step of obtaining data for the corresponding time period from the mode with no missing data and constructing a feature matrix for each mode with no missing data: Obtain the data for the corresponding time period from the mode with no missing data, and construct an initial matrix based on the data at each time point in the time period; The initial matrix is ​​convolved to obtain a feature matrix corresponding to each mode.

7. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 6 is characterized in that: In the step of constructing an initial matrix based on the data at each time point in the time period, the data at each time point in the time period is encoded, and the encoded data at each time point is combined to construct the initial matrix.

8. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 1 is characterized in that: In the step of fusing the feature matrices of multiple modalities to obtain a semantic consistency matrix, the feature matrices of multiple modalities are convolved to obtain the semantic consistency matrix.

9. The incomplete multimodal perception data recovery method based on conditional diffusion according to claim 4 is characterized in that: The method further comprises training optimization, and training is performed based on the following first loss function and second loss function: in, represents the value of the first loss function, represents the fusion matrix, The label feature matrix representing missing data, represents the trajectory matrix; in, represents the restored feature matrix, represents the value of the second loss function, Label feature matrix representing missing data.

10. An incomplete multimodal perception data recovery system based on conditional diffusion, characterized in that: The system includes a computer device, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 9.

Images