Cross-modal data completion and compression

Abstract

Aspects relate to analyzing a multi-modal data set using one or more cross-modal machine learning models. The machine learning model is operable to generate analysis data related to different data modalities. Analytical data may be used to identify related portions of data of different modalities. Once the relationships between the different modalities of the data are identified, the relationships may be used to perform a variety of different processes. For example, a first portion of data having a first modality may be used to reconstruct missing or erroneous data from a second modality. The relationships between the content stored in the different modalities may also be used to perform compression on the multi-modality data set.

Description

Background Art

[0001] The increase in the availability of devices that can use different modalities to transmit data has led to an increase in the amount of applications that use multimodal data. While applications used to primarily send data using a single modality (e.g., audio data for phone calls, text data for instant messaging), it is now common for applications to utilize different data modalities in a single session.

[0002] Aspects disclosed herein are presented with reference to these and other general considerations.In addition, although relatively specific problems are discussed, it should be understood that the examples should not be limited to solving specific problems identified in the background or elsewhere in this disclosure. Summary of the invention

[0003] Various aspects of the present disclosure relate to analyzing multimodal datasets using one or more cross-modal machine learning models. The machine learning models are operable to generate analysis data related to different data modalities. The analysis data can be used to identify relevant portions of data of different modalities. Once these relationships between different modalities of data are identified, these relationships can be utilized to perform a variety of different processes. For example, a first portion of data having a first modality can be used to reconstruct missing or erroneous data from a second modality. The relationships between content stored in different modalities can be further utilized to perform compression on the multimodal dataset.

[0004] This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the following description, and in part will be apparent from the description, or may be learned through practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0006] Figure 1 Depicted are exemplary systems for performing cross-modal data completion and / or compression.

[0007] Figure 2 An exemplary method for analyzing cross-modal data from one or more applications is described.

[0008] Figure 3 Depicted are exemplary methods for linking cross-modal datasets.

[0009] Figure 4Depicted are exemplary methods for reconstructing erroneous or missing content from a first data modality using one or more different data modalities.

[0010] Figure 5A Depicted are exemplary methods for utilizing analytical data from one or more cross-modal machine learning models and / or modality-specific machine learning models.

[0011] Figure 5B An exemplary method for decompressing session data.

[0012] Figure 6 is a block diagram illustrating example physical components of a computing device with which aspects of the present disclosure may be implemented.

[0013] Fig. 7A is a simplified diagram of a mobile computing device with which aspects of the present disclosure may be implemented.

[0014] Figure 7B is a simplified block diagram of another mobile computing device with which aspects of the present disclosure may be implemented. DETAILED DESCRIPTION

[0015] The various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which are from a portion of the present disclosure and illustrate specific example aspects. However, the different aspects of the present disclosure can be implemented in many different ways and should not be interpreted as being limited to the aspects set forth herein; on the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the various aspects to those skilled in the art. Practical aspects can be as methods, systems, or devices. Therefore, the various aspects can take the form of hardware implementation, completely software implementation, or a combination of software and hardware implementations. Therefore, the following detailed description should not be considered as limiting.

[0016] In addition to other benefits, aspects of the present disclosure utilize machine learning models to enable the use of data available in a second different modality to reconstruct data of a first modality. Aspects disclosed herein can be used to process multimodal data sets, and identify the minimum number of data points that can obtain data from different modalities at a certain point in time. For example, consider a multimodal data set for a movie (audio, visual video, transcription, etc.), and multiple data points for related video, audio, and text data are identified. These data points act as calibration points or reference points, and the number of calibration points can vary based on the characteristics of the multimodal content. In some aspects, calibration or reference points are used to cross-complement missing data from one modality based on another modality, for example, if audio is missing, audio can be interpolated and synthesized based on the data at the calibration points, and then the video or text modality is checked to generate audio for the missing part. In another aspect of the present disclosure, the data generated by the machine learning model can be used to significantly compress multimodal data. For example, in the case where the multimodal data is data of a person speaking, using calibration points for the person, aspects of the present disclosure can regenerate other images of the person speaking based on the text data of the interview.

[0017] Figure 1An exemplary system 100 for performing cross-modal data completion and / or compression is depicted. As shown in system 100, various different client devices (e.g., client device 104A and client device 104B) can interact with application server 106 and cross-modal machine learning service 102 via network 112. Network 112 can be a local area network, a wide area network, a cellular data network, the Internet, or any other type of network. Each client device (e.g., client device 104A and client device 104B) can execute one or more applications (e.g., (multiple) applications 108A executed on client device 104A and (multiple) applications 108B executed on client device 104B) as part of a session. For example, the session can be a communication session in which different data modalities (such as video, audio, and / or other types of data (e.g., text, images, etc.)) are shared between client devices 104A and 104B. In such an example, applications 108A and 108B can be communication applications. In another example, the session may be a game session in which client device 104A and client device 104B are participating. During the game session, different data modalities may be shared between client devices 104A and 104B, such as, for example, audio data (e.g., player communications), video data, graphic data, game data, text data (e.g., chat in a game), etc., shared between client devices 104A and 104B. In the example, applications 108A and 108B may be game applications. It will be appreciated by those skilled in the art that system 100 supports any type of application and / or session that shares multimodal data. As used herein, multimodal data includes various different data types (e.g., data modalities) that may be shared during a session. For example, different data types may include, but are not limited to, audio data, video data, text data, control data, game data, image data, enterprise data (e.g., documents, presentations, electronic forms, etc.), etc. It will be appreciated by those skilled in the art that system 100 is scalable so that any type of data modality may be processed by system 100. For example, application server 106 may be a communication server, a gaming server, or any other type of server used to facilitate communication or data transfer between client device 104A and client device 104B. Although a specific number of client devices and a single application server are depicted as part of system 100, those skilled in the art will appreciate that any number of client devices or application servers may be employed as part of system 100.

[0018] The cross-modal machine learning service 102 is operable to receive different data modalities transmitted between the client devices 104A, 104B and / or the application server 106 via the network 112. In an example, the cross-modal machine learning service 102 may require user consent from each participant in the communication session before receiving data from the session in order to maintain user privacy expectations. In one example, the cross-modal machine learning service 102 may receive data directly from the client devices 104A and 104B. Alternatively or additionally, the cross-modal machine learning service 102 may receive aggregated data for a session shared between the client devices from the application server 106. Figure 1 As shown, the cross-modal machine learning service 102 may include an input interface, one or more cross-modal machine learning models, a cross-modal data linker 118, a cross-modal data completion model 120, a compression process 122, a compressed data storage 124, and an output interface 126.

[0019] The input interface 114 is operable to receive application data associated with a session between client devices. In an example, the input interface can establish a communication channel with the client device 104A, the client device 104B and / or the application server 106 to receive application data transmitted as part of the session. As described above, the application data includes two or more different data modalities (e.g., data types). In another example, the input interface 114 is also operable to pre-process the application data to generate a feature vector set. The feature vector set can be provided as an input to one or more cross-modal machine learning models 116. In one example, a single feature vector set can be generated based on a representation of multimodal data. That is, using a session between communication applications as an example, the feature vector set can be generated based on the aggregation of different data modalities included in the application data received by the input interface 114 (e.g., the feature vector set can be generated based on a multimodal table of a session including video, audio, and text data). Alternatively or additionally, pre-processing application data can include generating different feature data sets for different types of data modalities. For example, application data can be parsed to identify different data modalities. Different data modalities in the application data can be aggregated, and the aggregated data for each modality can be processed to generate a feature vector set for each modality. In another example, one or more machine learning models can be used on the application data to separate different data modalities. The separated data modalities can be provided as outputs from one or more machine learning models and used to create different feature vector sets for the separated data modalities.

[0020] One or more feature sets may be provided to the cross-modal machine learning model 116 . Figure 1The cross-modal machine learning model(s) 116 are conceptually shown as different machine learning models designated to analyze different types of data modalities. For example, the cross-modal machine learning model(s) 116 include a natural language understanding (NLU) model 116A, an object detection model 116B, a video understanding model 116C, and an audio understanding model 116D. In one example, the cross-modal machine learning model(s) 116 can be a library of different machine learning models trained to process a specific type of data modality (e.g., Figure 1 ). Alternatively or additionally, (multiple) cross-modal machine learning models 116 can be a single machine learning model trained to process multiple data modalities at once. Conceptually, a single cross-modal machine learning model can include different layers for processing different types of data modalities (e.g., NLU layer, object detection layer, video understanding layer, audio understanding layer, etc.). NLU model 116A (or the NLU layer of a cross-modal machine learning model) can be trained to perform natural language understanding on spoken or textual communications. Object detection model 116B can be trained to identify different objects in application data, such as objects in images, presentations, documents, videos, etc. Video understanding model 116C can be trained to identify content in video data (such as images or text displayed in a video), perform lip reading to determine words spoken by people in the video, identify scene changes, background changes, or reference frames in the video, etc. These portions of the video data can be used as calibration points to reproduce video content. Video content can be labeled by video understanding model 116C based on content type. The audio understanding model 116D can be trained to identify audio content, such as spoken word, music, sound effects, etc. Audio content can be identified and labeled by the audio understanding model based on the content type (e.g., spoken word, music). The audio understanding model can be further trained to identify calibration points in the audio data, which can be used to reproduce the audio data (e.g., voice samples, music selections, etc.). In addition, the content of different models or layers within the same model can be provided to other models or other modality-specific layers in a single cross-modal model, and the content can be used as part of the analysis of content represented by other modalities. Although specific models and layers are described herein, those skilled in the art will understand that the cross-modal machine learning service 102 can be extended to perform other types of analysis on other types of data modalities. For example, the cross-modal machine learning service 102 may include: a lip reading model or (multiple) layers, which can be operated to determine the words spoken by the speaker based on lip reading a video feed of the speaker; an object character recognition model or layer to identify and interpret text displayed in application data; an action model, which is trained to process actions performed by characters in the game, and so on. Thus, the various aspects disclosed herein are extensible to operate on any type of application utilizing any different type of data modality.

[0021] The (multiple) cross-modal machine learning models 116 analyze the application data, for example, by processing one or more feature vector sets or processing raw application data. The analysis of the application data may include: identifying the content represented by different data modalities in the application data, marking the content in the application data, converting the data modality (e.g., using speech-to-text to convert voice communication), determining the intention for actions, statements, etc. In addition, the (multiple) cross-modal machine learning models 116 may identify one or more calibration points for different data modalities. As used herein, a calibration point may be part of a data modality that can be used to generate content in the same modality. For example, a reference frame of a video may be a calibration point. A reference frame may be used to generate other frames in a video stream. An audio sample of a user's voice (which may be used to reproduce the user's voice using a speech generator) may be a reference point. Those skilled in the art will appreciate that the calibration points identified by the (multiple) cross-modal machine learning models 116 may vary based on the type of data modality. In addition, the number of calibration points identified by the (multiple) cross-modal machine learning models 116 may vary depending on the type of underlying content represented by the data modality. For example, considering video content, if the video is relatively static (e.g., background does not change, few or no scene changes, etc.), then the (multiple) cross-modal machine learning model 116 may identify fewer data points than for dynamic video content (e.g., background changes, frequent scene changes, etc.) because fewer reference frames are required to reconstruct static video content than dynamic video content. Therefore, part of the analysis performed by the (multiple) cross-modal machine learning model 116 includes determining the content changes represented by different data modalities in order to determine which portions of the data of a particular modality should be marked as calibration points and stored to reproduce the content represented by that particular modality.

[0022] In some aspects, the cross-modal machine learning model can also be stored locally on the client device or application server. Figure 1 As depicted, client device 104A optionally stores and / or executes a local cross-modal machine learning model 108A, client device 104B optionally stores and / or executes a local cross-modal machine learning model 108B, and application server 106 optionally stores and / or executes a server cross-modal machine learning model 110. In some aspects, cross-modal data analysis can be performed locally to ensure that the user's private data remains on their device. In this case, the results of the analysis can be provided to the cross-modal machine learning service 102 in a manner that omits or obfuscates personal information so that the user cannot be identified based on the user's privacy preferences.

[0023] (multiple) cross-modal machine learning models 116 (and Figure 1The local and / or server CMML (as depicted) that may exist analyzes different data modalities to identify the content represented in the different modalities and the calibration points associated with the content for each modality. In this process, (multiple) cross-modal machine learning models 116 generate analysis data sets for different data modalities used by the application. In the example, the analysis data may include, but is not limited to: basic content represented by different data modalities or data related to the basic content, calibration points for different modalities, and data for cross-referencing related content from different modalities (e.g., timestamps, content descriptors, content identifiers, etc.). This cross-modal analysis data generated by (multiple) cross-modal machine learning models 116 can be used to perform error correction, generate missing data for one modality based on available data for different modalities, and / or perform compensation for cross-modal application data. To facilitate these processes, (multiple) cross-modal machine learning models 116 may include a cross-modal data linking process 118. The cross-modal data linking process 118 can be used to link corresponding parts of different data modalities. That is, the cross-modal data linking process 118 can utilize the cross-modal analysis data generated by the (multiple) cross-modal machine learning model 116 to determine corresponding portions of different data modalities related to the same or similar content. In one example, the corresponding portions can be determined based on a comparison of the analysis data for the different data modalities. The (multiple) cross-modal machine learning model 116 can generate representations of the underlying content for the different modalities in a manner that allows the content of the different data modalities to be compared. For example, a transcription of the user's utterances in the video can be generated based on a lip reading service by processing the video data (e.g., the first modality). Similarly, a transcription of the utterances can be generated based on audio data (e.g., the second modality) using a speech-to-text model or the speech-to-text layer of a single cross-modal machine learning model. The cross-modal data linking process 118 can compare the generated transcriptions to identify portions of the video data that correspond to portions of the audio data by identifying matching portions in the two transcriptions.

[0024] In another example, the cross-modal data linking process 118 can use metadata to link corresponding portions of different data modalities. For example, different data modalities may be associated with timestamps, content identifiers generated by (multiple) cross-modal machine learning models 116, content tags generated by (multiple) cross-modal machine learning models 116, or other types of metadata. In the example, the cross-modal data linking process 118 can use metadata in addition to comparing or not comparing the underlying content for different data modalities. In the example, the corresponding portions of different data modalities can be determined based on metadata to determine the portions of different data modalities that are generated, sent, or played simultaneously.

[0025] After determining relevant content for a first modality of data and a second, different modality of data, the cross-modal data linking process 118 generates data that links relevant portions of the data of the first modality with relevant portions of the data of the second modality. For example, the cross-modal data linking process 118 can generate reference data in the form of a data structure (e.g., a pointer, a reference table, etc.) that links corresponding portions of different modalities of data. The reference data can then be used in an error correction process, a missing data generation process, or a compression process, as will be described in further detail below. Although Figure 1 The cross-modal machine learning service 102 shown in FIG. 1 is depicted as having a separate cross-modal data linking process 118 , but in other examples, reference data or linking data may be generated by (multiple) cross-modal machine learning models 116 .

[0026] The cross-modal analysis data generated by the (multiple) cross-modal machine learning models 116 and the reference or link data generated by the cross-modal data linking process 118 can be provided to the cross-modal data completion model 120 and / or the compression process 122. The cross-modal data completion model 120 receives the analysis data, the link data and / or the original application data. The cross-modal data completion model receives the analysis data and / or the application data and determines whether one of the data modalities is missing content. As an example, consider a user speaking in a video call. At some point while the user is speaking, the user's microphone fails or is accidentally muted. In the example, the video data (e.g., the first modality) still shows the user speaking, however, the audio data (e.g., the second modality) will be missing content. The cross-modal data completion model 120 can analyze the application data or the analysis data and determine the missing portion of the audio data. The determination of the missing audio data can be performed in a variety of different ways. In one example, the cross-modal data completion model 120 can determine that the content for a specific data modality is missing for a period of time. In another example, the cross-modal data completion model 120 may compare corresponding analysis data from the first modality with the second modality data to determine that the second modality analysis data for the second modality data is missing, or to determine that the second modality analysis data for the second modality data is different from the analysis data for the first modality. In the example where the analysis data for the second modality is missing, it can be determined that the application data for the second modality is missing in the application data. When it is determined that the data for the second modality is missing, the cross-modal data completion model 120 can use the corresponding data from the first modality and the calibration points for the second modality to generate the missing data. For example, the cross-modal data completion model 120 can be trained to generate data of one or more modalities based on data representing content from different data types. Continuing with the example of missing audio data above, the analysis data generated by (multiple) cross-modal machine learning models 116 for a first data modality (e.g., video data) can be provided to the cross-modal data completion model 120 together with the calibration points generated by the cross-modal data completion model 120 for a second data modality (e.g., audio data) to generate the missing data for the second modality. For example, the analysis data for the video can include the transcription generated by the lip reading machine learning model. Based on the link data generated by the cross-modal data linking process, the portion of the transcription corresponding to the missing audio data can be provided to the cross-modal data completion model 120 to generate the missing audio data. In addition, the cross-modal data completion model 120 can use the calibration points for the audio data to generate spoken text that mimics the user's real voice for the transcribed portion.In this process, the missing data or otherwise damaged data for the second modality (e.g., audio data in the example) can be generated by using the calibration points for the second modality based on the analysis data for the second modality with the help of the cross-modal data completion model 120, and the analysis data represents the missing or damaged data (e.g., the voice of the simulated user in the example). The data generated by the cross-modal data completion model 120 can be provided to the output interface 126, which is operable to send the generated data to the requesting device (e.g., the client device 104A, the client device 104B and / or the application server 106). Additionally, the generated data can be provided to the compression process 122.

[0027] The compression process 122 receives analysis data from (multiple) cross-modal machine learning models 116, data from the cross-modal data linking process 118, and / or raw data from the session to perform compression of data related to the session. In addition to receiving data from the model, the compression process can also receive compression instructions. For example, the compression instructions can be to maximize compression to save storage space, minimize the computational requirements required for decompression and re-creation of multimodal data, set an acceptable level of quality loss based on compression, etc. Based on the instructions, the compression process uses the analysis data from (multiple) cross-modal machine learning models 116 and the data from the cross-modal data linking process 118 to determine which parts of the first data modality can be used to reconstruct parts of the second data modality. Once these related parts are identified, the compression process identifies parts of the original data that can be removed from the original data based on the compression instructions. For example, if the instruction is to maximize compression, data modalities in the original data that require larger storage space or are not easy to compress can be omitted from the compressed file. In contrast, calibration points for those modalities can be saved with the instructions to reconstruct the data using different modalities that are easy to compress or require less storage space during the decompression process. For example, the compressed file may not include video data, but may include calibration points for video data generated by (multiple) cross-modal machine learning models 116, and instructions for reconstructing the video data using data from different modalities and / or analysis data for video data or other data modalities. For example, when decompressing and providing session data, the calibration points can be retrieved from the compressed file and provided to the cross-modal data completion model 120 to reconstruct the original video data. Conversely, if the compression instructions are to reduce the processing required to reconstruct the original session data, more video data can be saved in the compressed file, and data (e.g., text data) that requires less computational processing can be omitted from the compressed file. The compressed file for the session can be stored in the compressed data storage 124. The output interface 126 can retrieve the compressed file and provide the compressed file to the requesting device. Before being transmitted to the requesting device via the output interface 126, the compressed file can be decompressed, and the omitted data type (e.g., video data from the above example) can be reconstructed, for example, using the cross-modal data completion model 120. Alternatively, if the requesting client device includes a local machine learning model capable of doing so, part or all of the compressed file for the session device may be decompressed and reconstructed locally on the requesting device.

[0028] Figure 2An exemplary method 200 for analyzing cross-modal data from one or more applications is depicted. In various aspects, the application can be any type of application that utilizes cross-modal data, such as, for example, a communication application, a video game, a media player, etc. That is, various aspects of the present disclosure can utilize different data types. For example, a communication application can be operable to generate, transmit, or play audio, video, and text information, and a video game can be operable to transmit graphics data, audio data, video data, text data, control instructions, etc. Those skilled in the art will appreciate that various aspects of the present disclosure can be practiced with any type of application that generates and / or provides various different types of cross-modal data.

[0029] The process begins at operation 202, where the method 200 receives application data. The application data can be received from one or more applications. In addition, the data can be received using a single device or multiple devices (such as Figure 1 The application data received at operation 202 includes cross-modal data, i.e., data with different modalities. For example, the application data may include text, audio, video, graphics, executable instructions, or any other type of data. As described above, aspects of the present disclosure are operable to process any type of data utilized by an application. The process continues to operation 204, where the received application data is preprocessed according to one or more cross-modal machine learning models. At operation 204, the application data may be processed to generate a feature vector, which may be provided as input to one or more cross-modal machine learning models. In one example, the application data may be received as a single stream or file including data with different modalities. In the example, a set of feature vectors may be generated based on a single stream or file. That is, the set of feature vectors may be generated based on a combined representation of different data modalities. Alternatively, the preprocessing operation may analyze a single file to individually identify different data modalities contained in the stream or file. For example, if an application data stream is a communication session that includes video, audio, and text (e.g., an instant messaging portion), the application data can be analyzed to independently identify different modalities. For example, the analysis can examine the contents of a stream or file to identify different data types. Alternatively, the data stream or data file can be processed using one or more machine learning models that are trained to identify and isolate different data modalities stored therein. Where the data modalities can be separated, in addition to or as an alternative to generating a feature set for a single data stream or file, separate feature vector sets can be generated for different data modalities.

[0030] At operation 206, one or more feature sets are analyzed using one or more cross-modal machine learning models. In one example, a single cross-modal machine learning model can be adopted to process one or more feature sets. In the example, the cross-modal machine learning model can be trained to identify and interpret different data modalities. That is, the cross-modal machine learning model can be trained to identify different data modalities (e.g., video, text, audio, natural language, etc.) from the feature vector set and interpret different data modalities. The cross-modal machine learning model can also be trained to interpret different data modalities. As used herein, interpreting different data modalities can include analyzing data modalities to mark, convert, determine intent, identify content, etc. Those skilled in the art will understand that the type of interpretation performed by the cross-modal machine learning model can vary based on the data modality, the type of application, the type of content being analyzed, or other considerations. That is, aspects of the present disclosure are extensible so that any type of analysis or intent processing known in the art can be performed at operation 206.

[0031] In an example, cross-modal data is interpreted to identify content represented by data of different modalities. That is, content for different modalities can be identified in a manner that can associate different modalities. For example, audio data can be processed using one or more cross-modal machine learning models to perform speech recognition. The recognized speech can be further processed using the natural language layer of the cross-modal machine learning model, or alternatively using another natural language understanding model, to generate a transcription of the spoken content. The transcription can be compared with other modalities (e.g., text chat, transcription generated using a lip reading machine learning model, etc.) to determine the relevant content of different data modalities.

[0032] Operation 206 may also include identifying one or more calibration points for different data modalities. As used herein, a calibration point may be part of the data, used to generate a representation of the data modality in the future. Take video data as an example. A calibration point may be a reference frame, which can be used to generate a portion of a video stream. Taking audio data being analyzed as an example, a calibration point may be a point in the audio, which can be used as a reference point to generate audio samples (e.g., samples of user voice, samples of background music, etc.) in the future. Those skilled in the art will appreciate that the number of different calibration points generated for different modalities varies based on the underlying content. Taking video as an example, in order to reproduce the video, the static part of the video data (e.g., a static background or a single scene) requires fewer calibration points than the dynamic part of the video (e.g., multiple scene changes). Therefore, the portion of the analysis performed at operation 206 may include determining how many calibration points are collected based on the underlying content. The location of the calibration point in the data may be identified at operation 206. Those skilled in the art will appreciate that a variety of different processes may be used to identify the calibration point based on the data modality associated with the calibration point.

[0033] When cross-modal data is analyzed, the process continues to operation 208, where the analyzed cross-modal data is linked. As described above, as part of operation 206, content represented by different data modalities is identified. At operation 208, links between different data modalities are determined, which identify related portions of the content of the different modalities. In one example, links can be created by comparing the identified content from different modalities to determine the similarities in the content. For example, a transcription generated based on spoken language in audio can be compared to a transcription generated by a lip-reading service using video data to identify similar portions of the transcriptions. Based on the comparison, a link can be generated that associates similar portions of the audio data and the video data. Additionally or alternatively, metadata associated with different modalities can be used to identify links between modalities. For example, if the modality includes a timestamp, the different data modalities can be linked by their timestamps. Further details on the linking process are described in Figure 3 is provided.

[0034] The process continues to operation 210, where the analyzed cross-modal data (e.g., content analysis, content identification, etc. generated by one or more cross-modal models), calibration points identified for different modalities, and link data are provided. In one aspect, providing the data can include storing the data for future use. The data generated at operation 206 can be used to perform compression of application data received at the operation, perform error correction, generate possible missing data from one or more modalities included in the original application data, etc. Therefore, the data can be provided to another process (such as a cross-modal data completion process to perform error correction or generate missing data), a compression process, or other applications.

[0035] Figure 3An exemplary method 300 for linking cross-modal data sets is depicted. The method 300 utilizes cross-modal analysis data to identify relevant portions in a cross-modal data set, and the cross-modal analysis data is generated using a cross-modal machine learning model and / or a machine learning model trained for a specific data type. For example, a cross-modal data set may include various different data types (e.g., video, audio, text, image, graphics, etc.) associated with a session (e.g., an application session, a communication session, a game session). The method 300 utilizes analysis data for different data modalities to determine corresponding portions of data of different modalities. For example, corresponding portions of data of different data modalities may be portions of a first modality and portions of a second modality related to the same content, such as portions of audio data of a user speaking corresponding to video data of the user speaking. The method 300 generates link or association data identifying corresponding data of different data modalities. The process begins at operation 302, where cross-modal analysis data is received from a cross-modal machine learning model or various machine learning models trained to analyze specific data modalities. In an example, the analysis data may include, but is not limited to, the base content represented by different data modalities or data related to the base content, calibration points for different modalities, data for cross-referencing related content from different modalities (e.g., timestamps, content descriptors, content identifiers, etc.). Alternatively or additionally, raw data for the session and / or metadata related to the session or each data modality are also included.

[0036] At operation 304, cross-modal analysis data and / or analysis data generated for each data modality are checked to determine identification points within the data. As used herein, identification points can be points in cross-modal data that are relevant to the underlying content. Alternatively or additionally, identification points can be other indicators within the data that can be used to reference the data, such as timestamps, labels, tags, or any other type of identification data. At operation 304, identification point sets for each data modality are generated (e.g., identification point sets for audio data, identification point sets for video data, identification point sets for text data, etc.). These identification points can be used to cross-reference underlying content from different data modalities to determine portions of different data types representing related content. As discussed above and herein, portions of different data types representing related content can be used to generate content for different modalities, compressed session content, etc.

[0037] At operation 306, the identification points from different data modalities are compared to determine whether the cross-modal data is related. For example, when the cross-modal data is related to the same basic content, they are related. For example, the video data of a person speaking during a communication session is related to the audio data of the person speaking. However, just having corresponding identification points in different data modalities does not mean that the content is related. Continuing with the example that the video data of a person speaking is related to the audio data of the person speaking, consider that the identification points for text data (e.g., representing the chat feature of a communication application) overlap with the video and / or audio data (e.g., with matching timestamps). The text data may be related to a separate conversation on the chat interface that occurs when the person speaks. In this instance, the text data may not be able to reconstruct the audio or video data. However, since the audio and video data are related to the same basic content, these two modalities can be used to reconstruct another modality. Therefore, at operation 306, the analysis data of two or more modalities with similar identification points are compared to determine whether the content is related. As described above, the (multiple) cross-modal machine learning models disclosed herein can generate analysis data that can be used to identify and / or interpret content stored in different data types. At operation 306, the analysis data for the different data types is compared to determine whether the different data types represent related content (eg, audio of a person speaking and corresponding video of the person speaking).

[0038] Based on the comparison performed at operation 306, at operation 308, two or more data modalities are linked. That is, when the content represented by two different modalities is related at the identification point, a link is created to associate the two data modalities at these points. In one example, the link is a pointer contained in one or both of the data modalities, pointing to the relevant part of the content of the other data modalities. In another example, a reference table storing the relationship between different data modalities can be generated. Those skilled in the art will appreciate that any type of data structure or reference type can be generated at operation 308. At operation 310, links and / or associations between different data modalities are provided. In one example, providing links and / or associations can include updating the analysis data for each data modality to include references to the relevant parts of the content of the other data modalities. In an alternative example, the links and / or associations can be provided as separate data elements, which are associated with the analysis data, the original session data and / or the calibration point.

[0039] Although method 300 is described as a process separate from the cross-modal data analysis performed by (multiple) cross-modal machine learning models, those skilled in the art will understand that in alternative examples, (multiple) cross-modal machine learning models may generate linked data when analyzing different data modalities.

[0040] Figure 4 An exemplary method 400 for reconstructing erroneous or missing content from a first data modality using one or more different data modalities (e.g., different data types) is depicted. In one example, the method 400 can be performed in real time or near real time. That is, when missing or erroneous data for a data modality is detected during a session, the missing data modality can be generated during the session using other data modalities. Alternatively, the method 400 can be performed for stored data of the session, for example, to generate missing or damaged portions of the first data modality based on data from one or more different data types. The process begins at operation 402, where multimodal data related to an application session or a communication session is received. As described above, aspects of the present disclosure can be performed on different types of applications (e.g., communication applications, games, enterprise applications, etc.). That is, the method 400 is operable to process any type of application or session including multimodal data. The multimodal data can be received in real time during a session. Alternatively, at operation 402, multimodal data from a completed session can be retrieved from a data store.

[0041] The process continues to operation 404, where analysis data is received from one or more cross-modal machine learning models and / or one or more machine learning models trained to analyze specific data types. As described herein, in addition to providing other information, analysis data for different data modalities also provides information about or interprets the content represented by different data modalities. After receiving multimodal data and / or analysis data, the process continues to operation 406, where it is determined whether the data of the first modality is wrong or missing. Various different techniques can be used to determine whether the data of the first modality is missing or wrong. For example, the data for the first modality can be analyzed to determine whether part of the data is missing (e.g., a part of the video is missing or does not exist in the multimodal data). As another example, the analysis data for the first modality can be compared with the analysis data for one or more different modalities to determine whether the data is missing or wrong. For example, the speech-to-text representation of audio data can be compared with the video-to-text representation of video data (e.g., generated using a lip-reading machine learning model or a lip-reading layer of a cross-modal machine learning model). If the text representations of the two modalities are the same, the data from the two modalities are likely to be complete and correct. However, if there are differences in the textual representations, there may be cases where the data from one of the modalities is missing or erroneous. In some aspects, confidence scores may be provided to the analyzed data. In this case, the analyzed data with the highest confidence score may be considered correct. Thus, points where the analyzed data for the other modality differs from the analyzed data with the highest confidence score may be identified as missing or erroneous.

[0042] After identifying one or more missing and / or erroneous portions of data of a first modality (e.g., having a first data type), the process continues to operation 408, where portions of data from a different modality (e.g., a different data type) corresponding to the missing or erroneous data of the first modality are identified. The identification performed at operation 408 can be based on link data (e.g., data generated by method 300 and / or by a cross-modal data linking process 118). After identifying corresponding portions from different data modalities, at operation 408, those corresponding data from different data modalities can be retrieved. At operation 410, one or more calibration points for the first data modality are retrieved. As described above, calibration points are used to generate the content of the modality to which they are associated. Therefore, the number of calibration points (or even whether calibration points are required) can vary depending on the data type of the first data modality and / or the underlying content of the first data modality.

[0043] At operation 412, the missing or corrupted data for the first data modality is reconstructed using corresponding portions of the data from the one or more different data modalities retrieved at operation 408 and possibly one or more calibration points for the first data modality. For example, at operation 412, a machine learning model trained to generate the first modality of data may be employed to generate the missing or erroneous data. Figure 1 The cross-modal data completion model 120 is one such exemplary model. The machine learning model can receive corresponding data from different modalities to generate the same content of the first modality. In addition, calibration points can be used by the machine learning model to generate content in a manner that simulates existing content of the first modality. For example, if the first modality is oral content, the calibration points can be used to generate oral audio in a voice similar to the user who spoke the content. As another example, if the missing or erroneous content is video content of a person speaking, calibration points from the video data (e.g., a reference frame) can be used to reconstruct the video of the person speaking. Without departing from the scope of the present disclosure, any technique known in the art for generating a simulation can be performed at operation 412.

[0044] After reconstructing the missing or erroneous data for the first modality, the reconstructed data is provided at operation 414. In one example, the reconstructed data can be provided in real time or near real time. That is, the reconstructed data can be provided during an ongoing session, thereby correcting erroneous or missing data while the session is ongoing. Alternatively or additionally, at operation 414, the reconstructed data can be stored with other session data.

[0045] Figure 5AAn exemplary method 500 for utilizing analysis data from one or more cross-modal machine learning models and / or modality-specific machine learning models is depicted. The process begins at operation 502, where multimodal data related to an application session or communication session is received. As described above, aspects of the present disclosure can be performed on different types of applications (e.g., communication applications, games, enterprise applications, etc.). Therefore, method 500 is operable to process any type of application or session that includes multimodal data. The multimodal data can be received in real time during the session. Alternatively, at operation 502, multimodal data from a completed session can be retrieved from a data store. The process continues to operation 504, where storage and / or performance requirements for the compression process are received. For example, the compression requirements can be to maximize compression to save storage space, minimize the computational requirements required to decompress and recreate the multimodal data, set an acceptable level of quality loss based on compression, etc. The process continues to operation 506, where analysis data is received from one or more cross-modal machine learning models and / or one or more machine learning models trained to analyze specific data types. As described herein, analysis data for different data modalities provides information about or interprets the content represented by the different data modalities, among other information. Such data can be used by the compression process to determine which parts of the different data modalities can be recreated using data from other modalities and which parts cannot.

[0046] At operation 508, the analysis data for different data modalities are compared, for example, using link data or comparing processes of various other types described herein. At operation 508, method 500 generates a list for different data modalities, listing which parts of the data modalities can be reconstructed and which other data modalities are used for reconstruction. At operation 510, the list is used to determine which parts of the different multimodal data related to the session should be stored in a compressed file for the session. In an example, storage and / or performance requirements are used to determine which parts of the session are to be stored in a compressed file. For example, if the requirements received at operation 504 indicate that the session data should be highly compressed, then for data types (such as video data and audio data) that require a large amount of storage in the session or cannot be highly compressed without losing acceptable quality, they can be removed from the compressed file for the session as much as possible. If the requirement is to minimize file size, then at operation 510, data modalities that require more storage space are analyzed to identify which parts of these modalities can be reconstructed using different modalities that require less storage space or can be highly compressed within an acceptable quality loss range. Since these portions can be reconstructed and doing so complies with compression requirements, these portions are identified by method 500 as not requiring storage.

[0047] After identifying the portion that does not require storage, the process continues to operation 512, where the associated calibration points, link data to other data modalities, and instructions for reconstructing the data of the identified portion are collected and / or generated. After collecting these data, the process continues to operation 513, where the data portions for different data modalities that can be reconstructed using the remaining data modalities are deleted or otherwise omitted from the compressed file for the session. At operation 514, the remaining session data, calibration points, link data to other data modalities, and instructions for reconstructing the deleted or otherwise omitted portions are compressed to generate a compressed file for the session. Those skilled in the art will appreciate that any compression process can be used at operation 514. The compressed file can then be stored in a compressed data store for the session and / or sent to one or more requesting devices for local storage on one or more compression devices.

[0048] Figure 5B 5 is an exemplary method 520 for decompressing session data. The process begins at operation 522, where compressed session data (e.g., a compressed file or data stream of a session) is received. At operation 524, the compressed session data is decompressed, for example, using an inverse process of the compression process used to compress the session data. Upon decompressing the session data, the process continues to operation 526, where missing data for one or more data modalities (e.g., data types) of the session data is identified. In one example, the techniques disclosed herein for identifying missing or corrupted data can be performed at operation 526. Alternatively or additionally, at operation 526, instructions stored in a session file to generate the missing data can be used to identify the missing portion.

[0049] The process continues to operation 528, where one or more calibration points for the missing data modality are accessed from the decompressed data store. In an example, calibration points for one or more different data modalities (e.g., for each missing data modality) can be retrieved from the decompressed session data. The process continues to operation 530, where the missing data for each missing modality is reconstructed using the retrieved calibration points and available data from other modalities of the multimodal session data. In an example, one or more machine learning models can be employed to reconstruct the missing data for each missing modality. In the example, a machine learning model trained to reconstruct data for a specific modality can be employed. In the example, at operation 530, a machine learning model for each missing modality can be used. Alternatively, a cross-modal machine learning model trained to reconstruct data for various modalities can be employed. In another example, a process similar to method 400 can be employed at operation 530. When reconstructing missing data for one or more modalities, at operation 532, the reconstructed data and the remaining decompressed session data are provided. For example, the decompressed and reconstructed session data may be provided for playback using an application or sent to a requesting device.

[0050] Figure 6 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 with which various aspects of the present disclosure may be practiced. The computing device components described below may be suitable for the computing device described above. In a basic configuration, the computing device 600 may include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of the computing device, the system memory 604 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 604 may include an operating system 605 and one or more program tools 606 suitable for performing various aspects disclosed herein. The operating system 605 may, for example, be suitable for controlling the operation of the computing device 600. In addition, various aspects of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and is not limited to any particular application or system. The basic configuration is Figure 6 608. The computing device 600 may have additional features or functionality. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage Figure 6 609 and the non-removable storage device 610.

[0051] As described above, a plurality of program tools and data files may be stored in system memory 604. When executed on at least one processing unit 602, program tools 606 (e.g., applications 620) may perform processes including, but not limited to, various aspects as described herein. Applications 620 may execute one or more cross-modal machine learning models 630, instructions for performing reconstruction of modal data 632, compression processes 634, and / or decompression processes 636, as described in greater detail herein. Other program tools that may be used according to aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, and the like.

[0052] In addition, aspects of the present disclosure may be implemented in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing a microprocessor, or circuits on a single chip containing electronic components or a microprocessor. For example, aspects of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 6 Each or many of the components shown in the can be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via a SOC, the functionality described herein regarding the capabilities of the client-to-switch protocol can be operated via application-specific logic integrated with other components of the computing device 600 on a single integrated circuit (chip). Various aspects of the present disclosure may also be practiced using other technologies capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, various aspects of the present disclosure may be practiced within a general-purpose computer or any other circuit or system.

[0053] The computing device 600 may also have one or more (multiple) input devices 612, such as a keyboard, a mouse, a pen, an audio or voice input device, a touch or sliding input device, etc. Output device (s) 614, such as a display, a speaker, a printer, etc. may also be included. The foregoing devices are examples, and other devices may be used. The computing device 600 may include one or more communication connections 616 that allow communication with other computing devices 650. Examples of communication connections 616 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0054] The term computer-readable medium as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media for storage of information (such as computer-readable instructions, data structures or program tools) implemented in any method or technology. System memory 604, removable storage device 609 and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassette, magnetic tape, disk storage or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 600. Any such computer storage medium may be a part of computing device 600. Computer storage media does not include carrier waves or other propagation or modulated data signals.

[0055] Communication media may be embodied by computer readable instructions, data structures, program tools, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal having one or more characteristics that are set or changed in some manner to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0056] Fig. 7A and 7B A computing device or mobile computing device 700 is shown with which aspects of the present disclosure may be practiced, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc. In some aspects, a client (e.g., such as a Figure 1 The client device 102 shown in the system 100 in FIG. 1 may be a mobile computing device. Fig. 7A, one aspect of a mobile computing device 700 for implementing various aspects is shown. In a basic configuration, the mobile computing device 700 is a handheld computer having both input elements and output elements. The mobile computing device 700 typically includes a display 705 and one or more input buttons 710 that allow a user to input information into the mobile computing device 700. The display 705 of the mobile computing device 700 can also be used as an input device (e.g., a touch screen display). If included as an optional input element, the side input element 715 allows further user input. The side input element 715 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 700 can incorporate more or fewer input elements. For example, in some aspects, the display 705 may not be a touch screen. In yet another alternative aspect, the mobile computing device 700 is a portable telephone system, such as a cellular phone. The mobile computing device 700 can also include an optional keypad 735. The optional keypad 735 can be a physical keypad or a "soft" keypad generated on the touch screen display. In various aspects, output elements include a display 705 for showing a graphical user interface (GUI), a visual indicator 720 (e.g., a light emitting diode), and / or an audio transducer 725 (e.g., a speaker). In some aspects, the mobile computing device 700 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 700 incorporates input and / or output ports for sending or receiving signals to or from external devices, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0057] Figure 7B 700 is a block diagram illustrating an architecture of one aspect of a computing device, a server, a cross-modal machine learning service, a mobile computing device, etc. That is, a computing device or mobile computing device 700 can be incorporated into a system 702 (e.g., a system architecture) to implement some aspects. The system 702 can be implemented as a "smart phone" capable of running one or more applications (e.g., a browser, email, a calendar, a contact manager, a messaging client, a game, and a media client / player). In some aspects, the system 702 is integrated into a computing device, such as an integrated digital assistant (PDA) and a wireless phone. In other aspects, the system 702 is a PC, a laptop computer, a server device, etc.

[0058] One or more application programs 766 may be loaded into the memory 762 and run on or in association with the operating system 764 for execution by the processor 760. The system 702 also includes a non-volatile storage area 769 within the memory 762. The non-volatile storage area 769 may be used to store persistent information that should not be lost if the system 702 loses power. The application programs 766 may use and store information in the non-volatile storage area 769, such as emails or other messages used by email applications. A synchronization application (not shown) also resides on the system 702 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep information stored in the non-volatile storage area 769 synchronized with corresponding information stored at the host computer. It should be understood that other applications may be loaded into the memory 762 and run on the mobile computing device 700 described herein.

[0059] System 702 has a power supply 770, which can be implemented as one or more batteries. Power supply 770 can also include an external power source, such as an AC adapter or a power docking station that replenishes or recharges the batteries.

[0060] The system 702 may also include a radio interface layer 772 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 772 facilitates wireless connections between the system 702 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 772 are performed under the control of the operating system 764. In other words, communications received by the radio interface layer 772 can be propagated to the application 766 via the operating system 764, and vice versa.

[0061] Visual indicator 720 (e.g., LED) can be used to provide visual notification, and / or audio interface 774 can be used to generate audible notification via audio transducer 725. In the configuration shown, visual indicator 720 is a light emitting diode (LED), and audio transducer 725 is a speaker. These devices can be directly coupled to power supply 770 so that when activated, they remain turned on for a period of time indicated by the notification mechanism, even if processor 760 and other components may be turned off to save power. LED can be programmed to remain turned on indefinitely until the user takes action to indicate the power-on state of the device. Audio interface 774 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to audio transducer 725, audio interface 774 can also be coupled to a microphone to receive audible input to promote telephone conversations. According to aspects of the present disclosure, the microphone can also be used as an audio sensor to promote the control of notifications, as described below. System 702 can also include a video interface 776, which enables the operation of the device connected to peripheral device port 730 to record still images, video streams, etc.

[0062] The mobile computing device 700 implementing the system 702 may have additional features or functionalities. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 7B This is shown by the non-volatile storage area 768.

[0063] Data / information generated or captured by the mobile computing device 700 and stored via the system 702 may be stored locally on the mobile computing device 700, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 772 or via a wired connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 700 via the radio interface layer 772 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transfer and storage components, including email and collaborative data / information sharing systems.

[0064] The description and illustration of one or more aspects provided in this application are not intended to limit or constrain the scope of the disclosure claimed for protection in any way. The disclosure claimed for protection should not be interpreted as being limited to any aspect or detail provided in the application, for example. Regardless of being shown and described in combination or individually, various features (both structure and method) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. Having provided the description and illustration of the application, those skilled in the art can envision changes, modifications and alternative aspects within the spirit of the broader aspects of the general inventive concept embodied in the application, which do not depart from the broader scope of the disclosure claimed for protection.

Claims

1. A system comprising: at least one processor; as well as A memory encoding computer executable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: receiving multimodal data associated with the session; providing the multimodal data to one or more machine learning models, wherein the one or more machine learning models are operable to generate first analysis data for at least a first modality of the multimodal data and second analysis data for at least a second modality of the multimodal data; determining, based on the first analysis data and the second analysis data, at least a first portion of data having the first modality, the first portion being capable of being reconstructed using at least a second portion of data having the second modality; determining one or more calibration points for the first modality of data, wherein the calibration points are operable to reconstruct the first portion of data in combination with the second portion of data; as well as The first analytical data, the second analytical data, and the one or more calibration points are provided.

2. A system according to claim 1, wherein the one or more machine learning models include a cross-modal machine learning model, and wherein a single cross-modal machine learning model is trained to analyze multiple different modalities including the multi-modality to generate analysis data for each of the multiple different modalities.

3. A system according to claim 2, wherein the cross-modal machine learning model is trained to process two or more of the following: video data, audio data, text data, image data, and graphic data.

4. The system of claim 1 , wherein the one or more machine learning models include two or more of: A first machine learning model trained to analyze video data; a second machine learning model trained to analyze the audio data; a third machine learning model trained to analyze text data; or A fourth machine learning model trained to analyze image data.

5. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations further comprising: First content having the first modality is linked to second content having the second modality, wherein the first content is related to the second content.

6. The system of claim 5, wherein linking the first content to the second content comprises: determining a first set of identification points for a first data set having the first modality; determining a second set of identification points for a second data set having the second modality; For corresponding identification points in the first identification point set and the second identification point set, comparing corresponding parts of the first analysis data and the second analysis data; as well as Based on the comparison, link data is generated.

7. The system of claim 6, further comprising instructions, which when executed by the at least one processor, cause the at least one processor to perform operations further comprising: determining a missing portion of data in the first data set having the first modality; determining, based on the linking data, a portion of data in the second data set that corresponds to the missing portion of data in the first data set; as well as The missing portion of data is reconstructed using the portion of data in the second data set that corresponds to the missing portion of data in the first data set and the one or more calibration points for the first modality of data.

8. The system of claim 6, further comprising instructions, which when executed by the at least one processor, cause the at least one processor to perform operations further comprising: determining an erroneous portion of data in the first data set having the first modality; determining, based on the linking data, a portion of data in the second data set that corresponds to the erroneous portion of data in the first data set; as well as The erroneous portion of the data is reconstructed using the portion of data in the second data set that corresponds to the erroneous portion of the data in the first data set and the one or more calibration points for the first modality of data.

9. A computer storage medium comprising computer executable instructions, which when executed by at least one processor, perform a method comprising: Receiving multimodal data; providing the multimodal data to one or more machine learning models, wherein the one or more machine learning models are operable to generate first analysis data for at least a first modality of the multimodal data and second analysis data for at least a second modality of the multimodal data; determining, based on the first analysis data and the second analysis data, at least a first portion of data having the first modality, the first portion being capable of being reconstructed using at least a second portion of data having the second modality; determining one or more calibration points for the first modality of data, wherein the calibration points are operable to reconstruct the first portion of data in combination with the second portion of data; determining missing portions of data in a first data set having the first modality; determining, based on the linking data, a portion of data in the second data set that corresponds to the missing portion of data in the first data set; as well as The missing portion of data is reconstructed using the portion of data in the second data set that corresponds to the missing portion of data in the first data set and the one or more calibration points for the first modality of data.

10. The computer storage medium of claim 9, wherein the multimodal data is associated with a communication application.

11. The computer storage medium of claim 10, wherein the method further comprises: determining a first set of identification points for a first data set having the first modality; determining a second set of identification points for a second data set having the second modality; For corresponding identification points in the first identification point set and the second identification point set, comparing corresponding parts of the first analysis data and the second analysis data; as well as Based on the comparison, link data is generated.

12. A method comprising: receiving multimodal data associated with the session; providing the multimodal data to one or more machine learning models, wherein the one or more machine learning models are operable to generate first analysis data for at least a first modality of the multimodal data and second analysis data for at least a second modality of the multimodal data; determining, based on the first analysis data and the second analysis data, at least a first portion of data having the first modality, the first portion being capable of being reconstructed using at least a second portion of data having the second modality; determining one or more calibration points for the first modality of data, wherein the calibration points are operable to reconstruct the first portion of data in combination with the second portion of data; providing the first analytical data, the second analytical data, and the one or more calibration points; generating instructions to reconstruct the first portion of data using the second portion of data; deleting said first portion of data; generating a file comprising the remaining multimodal data, the instructions to reconstruct the first portion of data, and the one or more calibration points; and Compress the file.

13. A method according to claim 12, wherein the one or more machine learning models include a cross-modal machine learning model, and wherein a single cross-modal machine learning model is trained to analyze multiple different modalities including the multi-modality to generate analysis data for each of the multiple different modalities.

14. The method of claim 12, wherein the one or more machine learning models include two or more of the following: A first machine learning model trained to analyze video data; a second machine learning model trained to analyze the audio data; a third machine learning model trained to analyze text data; or A fourth machine learning model trained to analyze image data.

15. The method of claim 12, wherein the multimodal data is associated with one of a communication session or a gaming session.

Images