System and method for performing multi-modal fusion

Abstract

Systems and methods for performing multi-modal fusion are provided. The method may include providing a data set including a plurality of rows of data. The multi-row data may include at least one row of unpaired modalities including the first modality and at least one row of paired modalities including both the first modality and the second modality. The method may further include interpolating, by a modality-specific encoder, at least one row of non-paired modalities by interpolating an embedding of a second modality from the paired modalities; training a modal specific encoder in the potential space based on interpolation of single-peak prediction and double-peak prediction; and generating confidence values for the single-peak prediction and the double-peak prediction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 452,638, filed on March 16, 2023, and U.S. Patent Application No. 18 / 339,075, filed on June 21, 2023, the disclosures of which are incorporated by reference in their entireties as if fully set forth herein. Technical Field

[0003] The present disclosure relates generally to deep learning. More specifically, the subject matter disclosed herein relates to systems and methods for multimodal fusion of missing and unpaired image and tabular data for defect classification. Background Art

[0004] The TV and mobile display industries have been growing rapidly in recent years. As new display panel modules and production methods are designed, enhanced equipment and quality control methods are desired to maintain quality during production because defects in display panels may result in excessive costs and losses. Closely monitoring the presence of defects that may occur during production can help reduce such costs and losses.

[0005] For example, some defects generated during production are repairable. In order to minimize the losses caused by these defects, repairable defects can be identified by using sensing equipment to collect certain data about the display panel. However, large-scale production and the microscopic size of potential defects make it difficult for human operators to find these defects. Instead, many modern technologies use computers to collect sensing data from various types of sensors to determine potential defects in the manufactured display. For example, sensors such as optical or infrared cameras can be used to capture images of various parts of the display panel, and computer vision can be used to identify certain features and / or characteristics of the display panel, and then determine whether the display panel is correctly and accurately produced according to manufacturing standards and tolerances. As another example, lasers can be used to measure various aspects of the display panel to ensure that the display panel is manufactured according to the expected size and within a certain tolerance. Accordingly, the computer can use this sensing data to automatically determine defects in production, so that such defects can be repaired or the conditions of the production process or machine that is causing such defects can be corrected to prevent further defects.

[0006] Accordingly, the sensory data collected for the display panel can generally consist of two modalities (images and tabular data). The image can be provided by a camera that captures, for example, a top view of the display panel, while the table records measurements (e.g., dimensions) and other relevant information. In order to effectively classify repairable defects and unrepairable defects, a multimodal classifier can be used to fuse information from the two modalities. In other words, the image is associated with the corresponding tabular data so that computer software using artificial intelligence (AI) can be trained to determine the presence of defects by using this fusion model. More specifically, some multimodal fusion techniques rely on complete and paired modalities, where each image is paired with the tabular data of the corresponding row. However, due to reasons such as lack of records or heterogeneous sources, it is usually difficult to obtain a modality-complete and paired data set for modality training. A common situation is that the data set includes only a small part of the modality complete and paired data and either images that lack tabular data or a mixture of the rest of the tabular data that lacks images. Therefore, it is not always guaranteed to have two modalities available for test samples. Therefore, there is a need for methods for performing multimodal fusion that can effectively learn from datasets with missing and / or unpaired modalities for high-accuracy classification and are robust to missing modalities at test time. Summary of the invention

[0007] According to an embodiment of the present disclosure, a method for performing multimodal fusion may include: providing a dataset including multiple rows of data, the multiple rows of data including at least one row of unpaired modality including a first modality and at least one row of paired modality including both the first modality and the second modality; interpolating at least one row of unpaired modality by a modality-specific encoder by interpolating an embedding of a second modality from the paired modality; training the modality-specific encoder in a latent space based on interpolation of unimodal predictions and bimodal predictions; and generating confidence values ​​for unimodal predictions and bimodal predictions.

[0008] Generation of confidence values ​​may include computing Shapley-based interpretations of unimodal and bimodal predictions.

[0009] The calculation based on Shapley's interpretation may include comparing the impact of the unimodal prediction and the impact of the bimodal prediction to a predetermined threshold.

[0010] The method may further include selecting a unimodal prediction or a bimodal prediction based on the generated confidence value.

[0011] The second modality may be a modality missing from the unpaired modality, wherein interpolation of the embedding comprises selecting K priors for the second modality from the K samples of at least one row of the paired modality, the K priors being selected as samples having an embedding that is closest to an embedding of the observed modality.

[0012] The method may further include: calculating a weighted sum of the K priors by performing cross-attention between the K samples and the K priors.

[0013] The first modality may correspond to a graphical modality and the second modality may correspond to a tabular modality.

[0014] The first modality may correspond to a tabular modality and the second modality may correspond to an image modality.

[0015] The method may further include training a modality specific encoder for the image modality by executing the visual transformer.

[0016] The method may further include training a modality-specific encoder for the tabular data modality by executing a feature-to-tokenizer transformer.

[0017] According to another embodiment of the present disclosure, a system for performing multimodal fusion may include: a memory; and a processor, configured to execute instructions stored in the memory to perform operations, the operations including: providing a data set including multiple rows of data, the multiple rows of data including at least one row of unpaired modality including a first modality and at least one row of paired modality including both the first modality and the second modality; interpolating at least one row of unpaired modality by a modality-specific encoder by interpolating an embedding of the second modality from the paired modality; training the modality-specific encoder in a latent space based on interpolation of unimodal predictions and bimodal predictions; and generating confidence values ​​for unimodal predictions and bimodal predictions.

[0018] Generation of confidence values ​​may include computing Shapley-based interpretations of unimodal and bimodal predictions.

[0019] The calculation based on Shapley's interpretation may include comparing the impact of the unimodal prediction and the impact of the bimodal prediction to a predetermined threshold.

[0020] The operation may further include selecting a unimodal prediction or a bimodal prediction based on the generated confidence value.

[0021] The second modality may be a modality missing from the unpaired modality, wherein interpolation of the embedding comprises selecting K priors for the second modality from the K samples of at least one row of the paired modality, the K priors being selected as samples having an embedding that is closest to an embedding of the observed modality.

[0022] The operation may further include calculating a weighted sum of the K priors by performing cross-attention between the K samples and the K priors.

[0023] The first modality may correspond to a graphical modality and the second modality may correspond to a tabular modality.

[0024] The first modality may correspond to a tabular modality and the second modality may correspond to an image modality.

[0025] The operations may further include training a modality specific encoder for the image modality by executing the visual transformer.

[0026] The operations may further include training a modality-specific encoder for the tabular data modality by executing a feature-to-tokenizer transformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the following section, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments illustrated in the accompanying drawings.

[0028] Figure 1 is an example of captured sensor data including images and tabular data according to an embodiment of the present disclosure.

[0029] Figure 2 is an example of sensor data through a multimodal deep learning model according to an embodiment of the present disclosure.

[0030] Figure 3 to Figure 4 They are respectively example block diagrams of the image pre-training stage and the table pre-training stage according to an embodiment of the present disclosure.

[0031] Figure 5 is a block diagram of an interpolation process for missing modalities according to an embodiment of the present disclosure.

[0032] Figure 6 is an example block diagram of a testing phase according to an embodiment of the present disclosure.

[0033] Figure 7 is an example block diagram of a framework for training and inference according to an embodiment of the present disclosure.

[0034] Figure 8 is a flow chart of a method for fusing missing and unpaired images and / or tabular modalities according to an embodiment of the present disclosure.

[0035] Fig. 9 is a block diagram of an electronic device in a network environment according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that the disclosed aspects may be implemented without these specific details. In other cases, well-known methods, processes, components and circuits are not described in detail to avoid obscuring the subject matter disclosed herein.

[0037] References to "one embodiment" or "embodiment" throughout this specification refer to specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases with similar meanings) appearing throughout this specification do not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described in this article as "exemplary" is not to be construed as necessarily superior to or advantageous over other embodiments. In addition, depending on the context discussed herein, a singular term may include a corresponding plural form, and a plural term may include a corresponding singular form. Similarly, hyphenated terms (e.g., "two-dimensional," "predetermined," "pixel-specific," etc.) may sometimes be used interchangeably with corresponding non-hyphenated versions (e.g., "two-dimensional," "predetermined," "pixel-specific," etc.), and capitalized entries (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock," "row select," "pixout," etc.). Such occasional interchangeable usage should not be considered inconsistent with each other.

[0038] It is also noted that the various drawings (including component drawings) shown and discussed herein are for illustrative purposes only and are not drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. In addition, reference numerals have been repeated between the drawings to indicate corresponding and / or similar elements, where deemed appropriate.

[0039] The terms used in this article are only used for the purpose of describing some example embodiments and are not intended to limit the claimed subject matter. As used in this article, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the / said" are intended to also include the plural forms. It should be further understood that when used in this specification, the term "includes" and / or its variations specify the presence of stated features, wholes, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or combinations thereof.

[0040] It will be understood that when an element or layer is referred to as being "on," "connected to," or "coupled to" another element or layer, it may be directly on, directly connected to, or coupled to another element or layer, or there may be intervening elements or layers. Conversely, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers. The same reference numerals refer to the same elements throughout. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0041] As used herein, the terms "first", "second", etc. are used as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. In addition, the same reference numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, this usage is only for simplicity of illustration and ease of discussion; it does not mean that the construction or architectural details of these components or units are the same in all embodiments or that these commonly referenced components / modules are the only way to implement some of the example embodiments disclosed herein.

[0042] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the subject matter belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless so explicitly defined herein.

[0043] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with the module. For example, software may be embodied as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any embodiment described herein may, for example, include, alone or in any combination, components, hardwired circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits. These modules may be embodied, collectively or individually, as circuits that form part of a larger system, such as, but not limited to, integrated circuits (ICs), systems on chips (SoCs), components, and the like.

[0044] Figure 1An example of captured sensor data including images and tabular data is illustrated. More specifically, some data rows include image modalities, some data rows include tabular data modalities (or tabular modalities), and some data rows include both image and tabular data modalities. Therefore, rows with both image and tabular data may be referred to as paired modalities or complete modalities. For the purposes of this disclosure, the terms "paired modality," "complete modality," and "completely paired modality" may be used interchangeably. On the other hand, some rows lack or are missing image modalities or tabular modalities. Such rows may be referred to as unpaired modalities or incomplete modalities. However, because a sensor does not necessarily always capture both image data and its corresponding tabular data, the captured sensor data results in many unpaired modalities.

[0045] If this captured sensor data is used as a model data set for a deep learning model, some technologies cannot utilize unpaired modalities. On the contrary, many technologies can only utilize paired modalities, and therefore may make unpaired modalities useless. Therefore, one way to achieve this requirement of paired modalities is by ignoring or discarding incomplete samples (i.e., unpaired modalities). However, this leads to a large amount of waste of data that has been collected. Another way to achieve this requirement is to replace the missing modality by interpolating zero or the average value. However, for complex modalities such as image modalities, this may be a difficult process. Another way to achieve this requirement is to generate the missing modality by the observed modality of the corresponding row (for example, if the tabular data modality is observed, the image modality is generated, or vice versa). However, the disadvantage of this technology is that it is heavily dependent on the correlation between the modalities, and therefore may be expensive and may be prone to introducing more noise than information. Therefore, it is necessary to waste less, less expensive and less noise-generated improved technology for utilizing a model data set with a missing modality.

[0046] As mentioned above, although sensor data often has missing modalities, many models are trained on both image modalities and tabular data modalities. Therefore, when such models are utilized during testing with sensor data having missing modalities, the result is that such models are not robust enough to handle such query samples. Figure 2 One such example is shown where a multimodal trained model is used with a query sample that has one row with the complete modality, one row with the tabular data modality missing, and one row with the image modality missing. In this case, even though the missing modality is imputed during testing, no quantitative explanation is generated.

[0047] Therefore, various embodiments of the present disclosure describe techniques for constructing an effective and efficient framework for training a multimodal defect classifier on data with missing or unpaired modalities. Such a framework can be constructed by training a model to efficiently generate representations of all data. Missing modalities can be efficiently interpolated by utilizing data that can be obtained from paired modalities. In doing so, the modality can be fused into paired modalities. Once the framework is created, robust predictions for test samples with missing modalities can be achieved by training the model to efficiently generate unimodal predictions and multimodal (e.g., bimodal) predictions. Accordingly, by constructing an effective framework and generating unimodal predictions and multimodal predictions, a quantitative explanation of the model predictions for test samples with missing modalities can be obtained. That is, quantitative measurements that can explain why and how two modalities (e.g., images and tabular data) are combined and confidence factors for the combined predictions can be generated.

[0048] Figure 3 to Figure 4 are example block diagrams of the image pre-training stage and the table pre-training stage according to various embodiments of the present disclosure. Figure 3 The image pre-training phase 306 in the image pre-training phase uses a reconstructed pre-trained image model (e.g., the visual transformer technique described in "Masked Autoencoders are Scalable Visual Learners" by He Kaiming et al. in the IEEE / CVF Computer Vision and Pattern Recognition 2022 Conference Proceedings (the contents of which are incorporated herein by reference in their entirety) as an encoder 308. Therefore, the encoder 308 can be trained to generate a representation or embedding of the image, and is therefore also referred to as an image encoder 308. In some embodiments, the visual transformer technique randomly masks 304 approximately 75% of the blocks of the original image 302, and the encoder 308 generates a prediction of the masked blocks at the encoder output 310. The decoder 312 can then be used to form an image 314 together to best represent what the original image 302 looks like. Accordingly, the image encoder 308 can be trained during this pre-training phase to reconstruct the masked input image. After the pre-training phase is completed, the encoder 308 is now trained, and the encoder 308 can now be used in operation (e.g., when operating a display panel defect classification system). In other words, mask 304 or decoder 312 are no longer utilized.

[0049] According to another embodiment, Figure 4The tabular pre-training stage in the training process combines the Scarf and Vime techniques disclosed in "Scarf: Self-supervised contrastive learning with random feature corruption" by Bahri Dara et al. in the International Conference on Learning Representations 2021 and "Vime: Extending the success of self-learning and semi-supervised learning to the tabular domain" by Yoon Jinsung et al. in Progress in Neural Information Processing Systems 33 (2020): 11033-11043 (the contents of which are incorporated herein by reference in their entirety) to train a tabular model, and a feature-tokenizer transformer is trained as a tabular encoder to generate a representation or embedding of the table. More details of the feature-tokenizer transformer technique are described in "Revisiting Deep Learning Models for Tabular Data" by Gorishniy Yury et al. in Progress in Neural Information Processing Systems 34 (2021): 18932-18943 (the contents of which are incorporated herein by reference in their entirety). Accordingly, randomly corrupted entries of table samples with values ​​are drawn according to their marginal probabilities, and a multi-level perceptual (MLP) decoder is used to learn by comparison and denoising. Therefore, if Figure 4 As shown in , an encoder 408, such as a feature-to-segmenter transformer, takes an original tabular input 402 that is randomly corrupted or has random feature corruption 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412. Figure 3 After this pre-training phase is complete, the encoder 408 (also referred to as a tabular encoder) is now trained and can now be used in operation (e.g., when operating a display panel defect classification system) (together with the image encoder 308). In other words, the random feature corruption 404 or the decoder 410 is no longer utilized.

[0050] After the pre-training phase is complete, the model encoders for the image modality and the tabular modality are ready for operations such as defect classification. Accordingly, when the encoder receives a dataset, for example from a sensor that collects raw information such as images and tabular data, the dataset may be missing modalities. That is, some rows of the dataset may include only image modalities or only tabular modalities, while some rows may include complete pairs containing both image modalities and tabular modalities. According to some embodiments, missing modalities can be interpolated in the latent space by interpolating existing embeddings of the same modality from existing complete pairs of modalities to interpolate missing embeddings instead of generating them from other observed modalities. In other words, interpolation is performed by interpolating the corresponding embedding of the same modality from another data row with a fully paired modality.

[0051] Figure 55 is a block diagram of a missing modality interpolation process according to some embodiments of the present disclosure. Here, the dataset 502 may be incomplete modality, because the image modality or the tabular modality may be missing. Therefore, if the sample lacks a certain modality, missing , K priors may be selected from the paired data. For example, if the tabular modality is the missing modality, K priors may be selected from the K samples 504 of the complete pairs of modalities to interpolate the embedding of the tabular modality from the K samples. In some embodiments, the K priors may be selected to have an embedding that is closest to the observed modality (i.e., the image modality) (this may be determined by the observed modality e observed Then, K prior embeddings of the missing modality (i.e., the tabular modality) with weights w can be calculated by cross-attention between the K samples and the K priors j This weighted sum can then be passed through a shallow (e.g., 2-layer) MLP j to generate an interpolated embedding at the tabular interpolation block 508. Finally, the interpolated embedding may be combined with the observed embedding to generate a concatenated embedding 512.

[0052] Similarly, if the image modality is the missing modality, K priors may be selected from the K samples 506 of the complete pairs of modalities to interpolate the embedding of the image modality from the K samples. In some embodiments, the K priors may be selected as having the embedding closest to the observed modality (i.e., the tabular modality) (this may be determined by the observed modality e observed Then, K prior embeddings of the missing modality (i.e., image modality) with weights w can be calculated by cross-attention between the K samples and the K priors j This weighted sum can then be passed through a shallow (e.g., 2-layer) MLP j to generate an interpolated embedding at the image interpolation block 510. Finally, the interpolated embedding may be combined with the observed embedding to generate a concatenated embedding 512.

[0053] After performing interpolation, each of the image encoder and the tabular encoder can be trained to perform both unimodal (e.g., unimodal) and bimodal (e.g., bimodal) predictions to achieve robustness to the missing modality. Accordingly, to do so, these encoders are tested during the testing phase to ensure that the unimodal predictions are reliable and also to ensure that the multimodal predictions are also reliable.

[0054] Figure 6604 and 606. Here, after extracting the representation of each modality, these representations can then be cascaded 608 and then provided to the prediction block 610. In some embodiments, the prediction block 610 can be a relatively shallow (e.g., 2-layer) transformer. In this way, the encoders 604, 606 can be fine-tuned together with the prediction block 610.

[0055] In some embodiments, the prediction block 610 (eg, transformer) may have three classification tags (CLS i ,i∈{I,T,I&T}), where I corresponds to image-based predictions, T corresponds to table-based predictions, and I&T corresponds to bimodal-based predictions. Therefore, during training, attention is masked so that unimodal classification labels only see labels in their corresponding modality, while bimodal classifications see all labels. These classification labels can be masked from each other. In some embodiments, the transformer can be trained with three cross-entropy losses to achieve robustness to missing modalities. More details are described in Ma Mengmeng et al., "Are Multimodal Transformers Robust to Missing Modalities?" in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2022 (which is incorporated herein by reference in its entirety).

[0056] Figure 7 is an example block diagram of a framework for training and inference according to various embodiments of the present disclosure. As discussed above, the prediction block has three classification labels, and the output from the prediction block is provided to the modality influence interpreter 702. In some embodiments, the impact of each modality i on the prediction v can be measured using a Shapley value by looking at the difference between making predictions with and without it. Accordingly, the modality influence interpreter 702 can generate a quantitative measure of the importance of each modality so that it can be determined how each modality affects the model, which in turn affects the predictions made by the model.

[0057] In some embodiments, the FastSHAP estimation technique described in "FastSHAP: Real-time Shapley Value Estimation" by Jethani Neil et al., International Conference on Learning Representations 2021 (which is incorporated herein by reference in its entirety) can be used to perform amortized estimation. Assuming the classification model is parameterized by θ, another multi-level perceptron (MLP) (β) can be trained as an estimator to obtain a surrogate value (φ′) for the Shapley value, where:

[0058]

[0059] The MLP can be trained to minimize the following loss:

[0060]

[0061] where m is a binary mask used to mask one of the two modalities, with probability Sample from {[1,0],[0,1],[1,1],[0,0]}.

[0062] Next, we can observe the influence of the mode and the effect of missing modalities It is compared with a confidence threshold τ and the final prediction is generated accordingly.

[0063]

[0064] Accordingly, the Shapley value can be used to determine the confidence values ​​of the unimodal prediction and the bimodal prediction, so that the prediction with the high confidence value can be selected and the encoder can use the prediction to fuse the missing unpaired image and tabular data. Accordingly, the computer can be programmed with a neural network configured to acquire raw image and tabular data from a sensor device at a manufacturing facility and automatically identify defects in a manufactured product (e.g., a display panel).

[0065] Figure 8802 is a flowchart of a method for fusing missing and unpaired image and / or tabular modalities according to various embodiments of the present disclosure. In some embodiments, a data set including multiple rows of data is provided. Each row of data may include columns of at least two modalities. According to an example, one of the modalities may be an image modality, and the other modality may be a tabular modality. Therefore, in an ideal case, a row of data will include both the image modality and the table modality. However, the technology provided in the present disclosure describes a method for providing a modality when one of the two modalities is missing. Therefore, in some embodiments, multiple rows of data may include at least one row of unpaired modalities and at least one row of paired modalities (step 802). In other words, the unpaired modality may include a first modality (which may be, for example, an image modality or a tabular modality) and the second modality may be missing, so the modality is unpaired in the row. The paired modality may include both the first modality and the second modality, for example, both the image modality and the tabular modality exist. Next, a modality-specific encoder can be used to interpolate at least one row of unpaired modalities by interpolating the embedding of the second modality from the paired modality (step 804). In other words, the missing modality, which is the second modality in this example, can be interpolated by interpolating existing data of the same modality as the missing modality from the available complete pairs of modalities. Next, a modality-specific encoder can be trained in the latent space based on the interpolation of unimodal predictions and bimodal predictions (step 806), and confidence values ​​for the unimodal predictions and bimodal predictions can be generated (step 808). In some embodiments, the confidence values ​​can be generated by a Shapley-based interpretation calculated as a quantitative measure corresponding to the importance of each modality. Accordingly, this interpretation quantifies how each modality can affect the model.

[0066] Fig. 9 is a block diagram of electronic devices in a network environment 900 according to an embodiment.

[0067] refer to Fig. 9, the electronic device 901 in the network environment 900 may communicate with the electronic device 902 via a first network 998 (e.g., a short-range wireless communication network), or communicate with the electronic device 904 or the server 908 via a second network 999 (e.g., a long-range wireless communication network). The electronic device 901 may communicate with the electronic device 904 via the server 908. The electronic device 901 may include a processor 920, a memory 930, an input device 950, a sound output device 955, a display device 960, an audio module 970, a sensor module 976, an interface 977, a haptic module 979, a camera module 980, a power management module 988, a battery 989, a communication module 990, a user identification module (SIM) card 996, or an antenna module 997. In one embodiment, at least one of the components (e.g., the display device 960 or the camera module 980) may be omitted from the electronic device 901, or one or more other components may be added to the electronic device 901. Some of the components may be implemented as a single integrated circuit (IC). For example, the sensor module 976 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be embedded in the display device 960 (eg, a display).

[0068] The processor 920 may execute software (eg, program 940 ) to control at least one other component (eg, hardware or software component) of the electronic device 901 coupled to the processor 920 , and may perform various data processing or calculations.

[0069] As at least part of data processing or calculation, the processor 920 may load a command or data received from another component (e.g., a sensor module 976 or a communication module 990) in the volatile memory 932, process the command or data stored in the volatile memory 932, and store the result data in the non-volatile memory 934. The processor 920 may include a main processor 921 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 923 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently of the main processor 921 or in conjunction with the main processor 921. Additionally or alternatively, the auxiliary processor 923 may be adapted to consume less power than the main processor 921, or to perform a specific function. The auxiliary processor 923 may be implemented as a separate part from the main processor 921 or as a part of the main processor 921.

[0070] The auxiliary processor 923 may replace the main processor 921 when the main processor 921 is in an inactive (e.g., sleep) state or control at least some of the functions or states related to at least one component (e.g., display device 960, sensor module 976, or communication module 990) among the components of the electronic device 901 together with the main processor 921 when the main processor 921 is in an active state (e.g., executing an application). The auxiliary processor 923 (e.g., an image signal processor or a communication processor) may be implemented as a part of another component (e.g., a camera module 980 or a communication module 990) that is functionally related to the auxiliary processor 923.

[0071] The memory 930 may store various data used by at least one component of the electronic device 901 (e.g., the processor 920 or the sensor module 976). The various data may include, for example, software (e.g., program 940) and input data or output data for commands related thereto. The memory 930 may include a volatile memory 932 and a non-volatile memory 934. The non-volatile memory 934 may include an internal memory 936 and / or an external memory 938.

[0072] The program 940 may be stored as software in the memory 930 , and may include, for example, an operating system (OS) 942 , middleware 944 , and applications 946 .

[0073] The input device 950 may receive a command or data to be used by another component (eg, processor 920) of the electronic device 901 from outside (eg, a user) of the electronic device 901. The input device 950 may include, for example, a microphone, a mouse, or a keyboard.

[0074] The sound output device 955 can output a sound signal to the outside of the electronic device 901. The sound output device 955 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or recording, and the receiver may be used to receive incoming calls. The receiver may be implemented as a part of the speaker or as a part of the speaker.

[0075] The display device 960 may visually provide information to the outside of the electronic device 901 (e.g., a user). The display device 960 may include, for example, a display, a holographic device, or a projector, and a control circuit that controls a corresponding one of the display, the holographic device, and the projector. The display device 960 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0076] The audio module 970 can convert sound into an electrical signal, and vice versa. The audio module 970 can obtain sound via the input device 950, or output sound via the sound output device 955 or an earphone of an external electronic device 902 directly (eg, wired) or wirelessly coupled to the electronic device 901.

[0077] The sensor module 976 may detect an operating state (e.g., power or temperature) of the electronic device 901 or an environmental state (e.g., user state) outside the electronic device 901, and then generate an electrical signal or data value corresponding to the detected state. The sensor module 976 may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0078] The interface 977 may support one or more specific protocols for direct (e.g., wired) or wireless coupling of the electronic device 901 with the external electronic device 902. The interface 977 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0079] The connection terminal 978 may include a connector via which the electronic device 901 may be physically connected to the external electronic device 902. The connection terminal 978 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

[0080] The haptic module 979 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via tactile or kinesthetic sense. The haptic module 979 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0081] The camera module 980 may capture still images or moving images. The camera module 980 may include one or more lenses, an image sensor, an image signal processor, or a flash. The power management module 988 may manage the power supplied to the electronic device 901. The power management module 988 may be implemented as at least a portion of a power management integrated circuit (PMIC), for example.

[0082] The battery 989 may supply power to at least one component of the electronic device 901. The battery 989 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0083] The communication module 990 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 901 and an external electronic device (e.g., electronic device 902, electronic device 904, or server 908) and performing communication via the established communication channel. The communication module 990 may include one or more communication processors that can operate independently of the processor 920 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. The communication module 990 may include a wireless communication module 992 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 994 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a first network 998 (e.g., such as Bluetooth TM The wireless communication module 992 can communicate with an external electronic device by using a standard short-range communication network such as a first network 998 or a second network 999 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single IC), or can be implemented as multiple components (e.g., multiple ICs) separated from each other. The wireless communication module 992 can use user information (e.g., an international mobile subscriber identity (IMSI)) stored in a user identification module card 996 to identify and authenticate the electronic device 901 in a communication network such as a first network 998 or a second network 999.

[0084] The antenna module 997 may transmit a signal or power to the outside of the electronic device 901 (e.g., an external electronic device), or receive a signal or power from the outside of the electronic device 901 (e.g., an external electronic device). The antenna module 997 may include one or more antennas, and thus, for example, at least one antenna suitable for a communication scheme used in a communication network such as the first network 998 or the second network 999 may be selected by the communication module 990 (e.g., the wireless communication module 992). Then, a signal or power may be transmitted or received between the communication module 990 and the external electronic device via the selected at least one antenna.

[0085] Commands or data may be sent or received between the electronic device 901 and the external electronic device 904 via a server 908 coupled to the second network 999. Each of the electronic devices 902 and 904 may be a device of the same type or a different type as the electronic device 901. All or some operations to be performed at the electronic device 901 may be performed at one or more of the external electronic devices 902, 904, and 908. For example, if the electronic device 901 should automatically or in response to a request from a user or another device to perform a function or service, the electronic device 901 may request one or more external electronic devices to perform at least a portion of the function or service instead of or in addition to performing the function or service. The one or more external electronic devices receiving the request may perform at least a portion of the requested function or service or an additional function or additional service related to the request, and transmit the result of the execution to the electronic device 901. The electronic device 901 may provide the result as at least a part of the reply to the request with or without further processing of the result. To this end, cloud computing, distributed computing, or client-server computing technology may be used, for example.

[0086] The subject matter and the embodiments of the operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware including the structures disclosed in this specification and their structural equivalents, or in one or more combinations thereof. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing device). The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded on the artificially generated propagation signal. The computer storage medium may also be, or may be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). In addition, the operations described in this specification may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0087] Although this specification may contain many specific implementation details, these implementation details should not be interpreted as limiting the scope of any claimed subject matter, but should be interpreted as a description of the specific features of a particular embodiment. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations, and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0088] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring the specific order shown or performing these operations in a sequential order or requiring the operations of all diagrams to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can usually be integrated together in a single software product or packaged in multiple software products.

[0089] Thus, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results.

[0090] As those skilled in the art will recognize, the innovative concepts described herein can be modified and varied in a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any specific exemplary teachings discussed above, but rather is defined by the appended claims.

Claims

1. A method for performing multimodal fusion, comprising: providing a data set including a plurality of rows of data, the plurality of rows of data including at least one row of unpaired modalities including a first modality and at least one row of paired modalities including both the first modality and a second modality; interpolating, by a modality-specific encoder, the at least one row of the unpaired modality by interpolating an embedding of the second modality from the paired modality; training said modality specific encoder in a latent space based on said interpolation of unimodal predictions and bimodal predictions; and Confidence values ​​for the unimodal prediction and the bimodal prediction are generated.

2. The method according to claim 1, wherein: The generating of the confidence value comprises computing a Shapley-based interpretation of the unimodal prediction and the bimodal prediction.

3. The method according to claim 2, wherein: The calculation based on Shapley's interpretation includes comparing the impact of the unimodal prediction and the impact of the bimodal prediction to a predetermined threshold.

4. The method according to claim 3, further comprising: The unimodal prediction or the bimodal prediction is selected based on the generated confidence value.

5. The method according to claim 1, wherein: the second mode is the mode missing from the unpaired modes, Therein, the interpolation of the embedding comprises selecting K priors of the second modality from the K samples of the at least one row of paired modalities, the K priors being selected as samples having an embedding that is closest to the embedding of the observed modality.

6. The method according to claim 5, further comprising: A weighted sum of the K priors is calculated by performing cross-attention between the K samples and the K priors.

7. The method according to any one of claims 1 to 6, wherein: The first modality corresponds to a graphical modality and the second modality corresponds to a tabular modality.

8. The method according to any one of claims 1 to 6, wherein: The first modality corresponds to a tabular modality and the second modality corresponds to an image modality.

9. The method according to any one of claims 1 to 6, further comprising: The modality-specific encoder is trained for the image modality by executing a visual transformer.

10. The method according to any one of claims 1 to 6, further comprising: The modality-specific encoder is trained for the tabular data modality by executing a feature-to-tokenizer transformer.

11. A system for performing multimodal fusion, comprising: Memory; as well as A processor configured to execute instructions stored in the memory to perform operations, the operations comprising: providing a data set including a plurality of rows of data, the plurality of rows of data including at least one row of unpaired modalities including a first modality and at least one row of paired modalities including both the first modality and a second modality; interpolating, by a modality-specific encoder, the at least one row of the unpaired modality by interpolating an embedding of the second modality from the paired modality; training said modality specific encoder in a latent space based on said interpolation of unimodal predictions and bimodal predictions; and Confidence values ​​for the unimodal prediction and the bimodal prediction are generated.

12. The system according to claim 11, wherein: The generating of the confidence value comprises computing a Shapley-based interpretation of the unimodal prediction and the bimodal prediction.

13. The system according to claim 12, wherein: The calculation based on Shapley's interpretation includes comparing the impact of the unimodal prediction and the impact of the bimodal prediction to a predetermined threshold.

14. The system according to claim 13, wherein: The operations further include selecting the unimodal prediction or the bimodal prediction based on the generated confidence value.

15. The system according to claim 11, wherein: the second mode is the mode missing from the unpaired modes, Therein, the interpolation of the embedding comprises selecting K priors of the second modality from the K samples of the at least one row of paired modalities, the K priors being selected as samples having an embedding that is closest to the embedding of the observed modality.

16. The system of claim 15, wherein: The operations further include computing a weighted sum of the K priors by cross-attention between the K samples and the K priors.

17. A system according to any one of claims 11 to 16, wherein: The first modality corresponds to a graphical modality and the second modality corresponds to a tabular modality.

18. A system according to any one of claims 11 to 16, wherein: The first modality corresponds to a tabular modality and the second modality corresponds to an image modality.

19. The system according to any one of claims 11 to 16, wherein: The operations further include training the modality specific encoder for an image modality by executing a visual transformer.

20. The system according to any one of claims 11 to 16, wherein: The operations further include training the modality-specific encoder for a tabular data modality by executing a feature-to-tokenizer transformer.

Images