Fusing multi-modal data using recurrent neural networks

Abstract

The embodiments relate to systems, program products, and methods for employing deep learning techniques to fuse data across modalities. A multimodal dataset is received, comprising a first dataset having a first modality and a second dataset having a second modality, wherein the second modality differs from the first modality. The first and second datasets are processed, including: encoding the first dataset into one or more first vectors, and encoding the second dataset into one or more second vectors. The processed multimodal dataset is analyzed, and the encoded features from the first and second modalities are iteratively and asynchronously fused. The fused modality comprises a combined vector representing relevant temporal behavior from the first and second datasets. The fused vector is then returned as output data.

Description

Technical Field

[0001] This embodiment relates to an artificial intelligence platform configured to fuse data across two or more modalities. More specifically, given a multimodal dataset having at least one modality representing numerical time series data and at least one modality representing timestamped text data, the embodiment relates to automatically discovering a subset of textual features that strongly correlate the performance and behavior of the numerical time series data as it evolves over time. Background Technology

[0002] Data relevant to almost every discipline comes from a variety of different sources in various formats and modalities. Recent deep learning techniques utilize information from multiple modalities and formats to create models that perform a variety of tasks, such as image annotation, medical diagnosis, and treatment recommendations. Various techniques for fusing data from multiple modalities have been proposed in the literature.

[0003] Three common fusion techniques are (a) early fusion (at the data feature level), (b) late fusion (at the decision level), and (c) intermediate fusion (a trade-off between (a) and (b)). Most fusion models require data from different modalities to be correctly aligned in timestamps or intervals in order to extract useful features across multiple modalities for a given objective. Aligning all modalities may be impractical due to data granularity, data sampling, or the nature of the data, and prior knowledge about the modalities is also assumed for proper alignment. Therefore, addressing these issues is necessary in this field. Summary of the Invention

[0004] From a first aspect, the present invention provides a computer system comprising: a processor operatively coupled to a memory; an artificial intelligence (AI) platform communicating with the processor, having a machine learning (ML) tool employing deep learning techniques to fuse data across modalities, the tool comprising: a first data feed manager operatively coupled to a first dataset having a first modality employing a first data format; a second data feed manager operatively coupled to a second dataset having a second modality employing a second data format, the second modality being different from the first modality; the first data feed manager encoding the first dataset into a first vector set; the second data feed manager encoding the second dataset into a second vector set; an analyzer operatively coupled to the first and second data feed managers, the analyzer utilizing an artificial recurrent neural network (RNN) to analyze the encoded first and second datasets, including iteratively and asynchronously fusing encoded features from the first and second data modalities, the fusing comprising combining vectors representing related temporal behavior from the first and second datasets; and the fused vectors being returned as output data.

[0005] In another aspect, the present invention provides a computer program product employing deep learning technology to fuse data across modalities. This computer program product includes a computer-readable storage medium embodying program code, which is executable by a processor to: receive a multimodal dataset comprising data from two or more modalities in different formats, including a first dataset having a first modality and a second dataset having a second modality; process the first and second datasets respectively, including: encoding the first dataset into one or more first vectors and encoding the second dataset into one or more second vectors; analyze the processed multimodal dataset, including iteratively and asynchronously fusing encoded features from the first and second data modalities, the fusion modality including combining vectors representing correlated temporal behavior from the first and second datasets; and return the fused vectors as output data.

[0006] In another aspect, the present invention provides a method comprising: receiving a multimodal dataset by a computing device, the multimodal dataset comprising data from two or more modalities in different formats, including a first dataset having a first modality and a second dataset having a second modality; processing the first dataset and the second dataset respectively, including: encoding the first dataset into one or more first vectors and encoding the second dataset into one or more second vectors; analyzing the processed multimodal dataset, including iteratively and asynchronously fusing encoded features from the first data modality and the second data modality, the fusing comprising combining vectors representing related temporal behavior from the first dataset and the second dataset; and returning the fused vectors as output data.

[0007] In another aspect, the present invention provides a computer program product for employing deep learning technology to fuse data across modalities. The computer program product includes a computer-readable storage medium that can be read by processing circuitry and stores instructions for execution by the processing circuitry to perform a method for performing the steps of the present invention.

[0008] In another respect, the present invention provides a computer program that is stored on a computer-readable medium and can be loaded into the internal memory of a digital computer, including a software code portion that, when the program is run on the computer, performs the steps of the present invention.

[0009] Examples include systems, computer program products, and methods for using an artificial intelligence platform to fuse data across modalities to identify correlations between text-based data and numerical time series data.

[0010] In one aspect, a computer system supporting an artificial intelligence (AI) platform is provided. As shown, a processor is operatively coupled to memory and communicates with the AI ​​platform. The AI ​​platform is equipped with tools employing deep learning techniques to fuse data across modalities. These tools include a first data feed manager, a second data feed manager, and an analyzer. The first data feed manager is operatively coupled to a first dataset representing a first modality using a first data format. The second data feed manager is operatively coupled to a second dataset representing a second modality using a second data format. The first and second data modalities are distinct. The first data feed manager encodes the first dataset into a first set of vectors. The second data feed manager encodes the second dataset into a second set of vectors. The analyzer is operatively coupled to both the first and second data feed managers and is used to analyze the encoded first and second datasets using an artificial recurrent neural network. The analyzer iteratively and asynchronously fuses the encoded features from the first and second data modalities, wherein the fusion combines vectors from the first and second datasets that represent related temporal behavior. The fused vectors are then returned as output data.

[0011] On the other hand, a computer program product employing deep learning techniques to fuse data across two or more modalities is provided. The computer program product includes a computer-readable storage medium having program code executable by a processor. Program code is provided for receiving two or more datasets, each dataset representing a different data modality, including a first dataset having a first modality and a second dataset having a second modality. The program code processes the first and second datasets respectively. The first dataset is encoded into one or more first vectors, and the second dataset is encoded into one or more second vectors. The processed multimodal datasets are analyzed, and the encoded features from the first and second modalities are iteratively and asynchronously fused. The fused modality includes a combined vector representing the relevant temporal behavior from the first and second datasets. The program code then returns the fused vectors as output data.

[0012] In another aspect, a method is provided that employs deep learning techniques to fuse data across two or more modalities. A multimodal dataset is received, comprising a first dataset having a first modality and a second dataset having a second modality. The first modality differs from the second modality. The first and second datasets are processed, including encoding the first dataset into one or more first vectors and encoding the second dataset into one or more second vectors. The processed multimodal dataset is analyzed, and the encoded features from the first and second modalities are iteratively and asynchronously fused. The fused modality comprises a combined vector representing the relevant temporal behavior from the first and second datasets. The fused vector is then returned as output data.

[0013] These and other features and advantages will become apparent from the following detailed description of the presently preferred embodiments, taken in conjunction with the accompanying drawings. Attached Figure Description

[0014] The reference numerals in the accompanying drawings form part of the specification. The features shown in the drawings are merely illustrative of some embodiments, and not of all embodiments, unless otherwise expressly indicated.

[0015] Figure 1 A diagram illustrating the architecture of a neural network is shown.

[0016] Figure 2 The description is in Figure 1 A diagram showing the memory units of the AsyncLSTM embedded in the neural network structure.

[0017] Figure 3 A flowchart illustrating the fusion of multimodal data using a recurrent neural network is shown.

[0018] Figure 4 A system diagram illustrating a computer system with an integrated artificial intelligence platform that supports the fusion of multimodal data is shown.

[0019] Figure 5 It describes the application of this method in the financial market. Figure 1 A block diagram of the neural network architecture.

[0020] Figure 6 The description is as follows Figure 4 The diagram shown illustrates the artificial intelligence platform tools and their associated application programming interfaces (APIs).

[0021] Figure 7 A block diagram illustrating an example of a cloud-based supporting computer system / server is provided to achieve the above-mentioned... Figures 1-6 The system and process described.

[0022] Figure 8A block diagram illustrating a cloud computing environment is provided.

[0023] Figure 9 A block diagram is drawn illustrating a set of functional abstraction model layers provided by a cloud computing environment. Detailed Implementation

[0024] It is readily understood that, as generally described and illustrated in the accompanying drawings, the components of this embodiment can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of embodiments of the apparatus, system, method, and computer program product of this embodiment, as presented in the drawings, is not intended to limit the scope of the claimed embodiments, but merely represents selected embodiments.

[0025] Throughout this specification, references to "selected embodiment," "an embodiment," or "embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "selected embodiment," "in an embodiment," or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment.

[0026] The illustrated embodiments will be better understood by referring to the accompanying drawings, in which the same components are always designated by the same reference numerals. The following description is intended to be illustrative only and shows only some selected embodiments of devices, systems, and processes consistent with the embodiments claimed herein.

[0027] Artificial intelligence (AI) is a field of computer science that encompasses computers and their human-related behavior. AI refers to the intelligence of machines when they can make decisions based on information, maximizing their chances of success on a given topic. More specifically, AI can learn from datasets to solve problems and provide relevant recommendations. For example, in the field of artificial intelligence computer systems, natural language systems (such as those from IBM...) Artificial intelligence computer systems or other natural language question-answering systems process natural language based on the knowledge acquired by the system. To process natural language, the system can be trained with data derived from databases or knowledge bases; however, the results may be incorrect or inaccurate for various reasons. IBM and IBM Watson are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.

[0028] Machine learning (ML) is a subset of AI that utilizes algorithms and corresponding neural networks to learn from data and create predictions based on that data. More specifically, ML is the application of AI through the creation of neural networks that can demonstrate learning behavior by performing tasks that are not explicitly programmed. Deep learning is a type of neural network ML where systems can accomplish complex tasks by using multiple layers of selection based on the output of previous layers, thereby creating increasingly intelligent and abstract conclusions.

[0029] Deep learning is a machine learning method that incorporates neural networks into successive layers to learn from data iteratively. A neural network is a model of how a nervous system operates. The basic unit is called a neuron, which is typically organized into layers. Neural networks work by mimicking a large number of interconnected processing units that are abstract versions of neurons. A neural network typically has three parts: an input layer with units representing the input field, one or more hidden layers, and an output layer with one or more units representing the target field(s). These units are connected with varying connection strengths or weights. Input data is fed to the first layer, and values ​​propagate from each neuron to each neuron in the next layer. At a basic level, each layer of a neural network includes one or more operators or functions that are operationally coupled to the input and output. The output of the operator(s) or function(s) from the last hidden layer is referred to here as the activation. Finally, the result is delivered from the output layer. Deep learning complex neural networks are designed to mimic how the human brain works, thus allowing computers to be trained to support poorly defined abstractions and problems. Neural networks and deep learning are frequently used in image recognition, speech recognition, and computer vision applications.

[0030] AI, especially deep learning, has made significant progress in many fields, such as autonomous driving, machine translation, and speech recognition, with profound implications for our society. Recurrent Neural Networks (RNNs) are artificial neural networks used in Natural Language Processing (NLP). RNNs are designed to recognize sequential characteristics of data and use patterns to predict outcomes. As shown and described in this paper, RNNs are constructed using modified Long Short-Term Memory (LSTM) units to classify data into short-term and long-term memory units. The modified LSTM described in detail below is called an Asynchronous RNN (AsyncLSTM), which iteratively fuses encoded features from multiple different data modalities, such as time-series data modalities and text data modalities. The output from the Asynchronous RNN includes an interpretation of the time-series behavior.

[0031] refer to Figure 1 Figure (100) is provided as an illustration of the neural network architecture. As shown, the model receives data from two input feeds, shown here as a first feed (110), e.g., feed 0, and a second feed (120), e.g., feed 1. The data filling the first feed (110) represents the first feed mode, and the data filling the second feed (120) represents the second mode, wherein the first and second modes are distinct. For descriptive purposes, the first feed (110) includes text data, and the second feed (120) includes time-series data; however, the number of feed modes should not be considered limiting.

[0032] As shown in the figure, the first feed (110) is operatively coupled to a text module (112) configured for natural language processing (NLP). The text module (112) learns the semantic dependencies between words present in the first feed (110) and aggregates them into a representative vector for each document, or, in one embodiment, a representative vector for each news story. The input to the text module (112) at each time period t is a sequence of news stories or articles n. Each news story or article is, in turn, a sequence of words and is analyzed and encoded separately. The j-th news story is a sequence Where K is the sequence length. V represents the vector representation of the l-th word in the sequence (l = 1, ..., K), and V represents the vocabulary size. The embedding layer is used to transform each vector via the following linear transformation... Transform into a low-dimensional dense vector So that The transformation reduces the word dimension from the vocabulary size V to d. w , where d w <<V. A set of representative vectors (114) is shown here as the output from the text module (112). Each vector represents a text-based document, such as a news article or document. In one embodiment, the representative vectors (114) are ordered sequentially based on the corresponding timestamp associated with each vector. For example, in one embodiment, each vector is a news article with a corresponding timestamp identifying the publication or release of the article.

[0033] Each embedding word sequence {e1,e2,…e} for a text article or document (110) K The input is fed into a text module (112), which is implemented by a single- or multi-layer LSTM (or Bi-LSTM) network (112) that learns to generate encoded contextual features or representative vectors for the text article used in the document (110). The network (112) explores contextual dependencies between ordered words to learn the optimal document representation. One of the main components of the LSTM unit is a memory unit that stores basic information about the input sequence over time, as well as a non-linear gating unit that regulates the flow of information between the input and output units. At each step l in the input sequence, the LSTM receives the input word embedding e l Its previous unit state and the previous output vector To update memory units And then output the response to e l The word indicates Where d h This represents the number of hidden neurons. Therefore, LSTM can be represented by the following recursive function f:

[0034] For l = 1, ..., K

[0035] Among them, memory unit It is updated internally. This network component is applied to each news story collected over time period t and outputs a vector sequence of news. t ={s1,s2,…s n}, where n is the total number of news stories collected at timestamp t. Each vector represents a sequence of text articles (such as news stories) or related words.

[0036] The AsyncLSTM (130) is configured to manage multiple input sequences of varying lengths. As shown here, the AsyncLSTM (130) utilizes inputs from two modalities, including an input (116) pointing to a sequence of embedded text documents (e.g., news stories) from a first input feed 0 (110) and an input (122) of numerical time series data from a second input feed 1 (120). In one embodiment, the first input feed 0 (110) points to news stories, encoded as "news" in a representative vector. t ={s1,s2,…s n The AsyncLSTM (130) iteratively fuses encoded features from different data modalities in an asynchronous manner. When modeling time series, the AsyncLSTM (130) performs multiple steps related to the text modality to progressively filter out news vectors that are irrelevant to the patterns learned in the time series, while retaining news vectors that are well aligned with the current behavioral state or performance of the time series data. The encodings (114) from the text data and the encodings (132) from the time series data are combined in a cascaded layer (150) to aggregate the encodings into a single vector, which is then passed through a multilayer perceptron network (160) that produces an output (170) in the form of behavioral labels for the time series data for a defined time period.

[0037] refer to Figure 2 Figure (200) is provided as an illustration of the memory cell (280) of the AsyncLSTM. Figure 1The AsyncLSTM receives two data modalities: an embedded text story sequence (also referred to herein as text data) from a first input feed 0 (110) and an input (122) of numerical time series data from a second input feed 1 (120). The text modality and the time series modality are sampled at different frequencies, resulting in different sequence lengths. Furthermore, the text is general and can introduce noise into the model. The AsyncLSTM is designed and configured to process multiple input sequences of varying lengths. The model shown and described herein leverages the interaction between the two data modalities through multiple interconnected steps between representative text vectors and time series time steps. As the AsyncLSTM progresses through the time series, it progressively filters out news vectors irrelevant to the patterns learned in the time series and obtains attention probability quality for the news stories that best match the time series.

[0038] The inputs to the memory unit (280) include time-series data from the second input feed 1 (220) and text data from the first input feed 0 (210). Regarding the time-series data, the input sample (220) at each time point t is represented as... The sequence of m values ​​of (222) and the previous hidden state (224). Similarly, input samples (212, such as news) t The sequence is represented as {s1, s2, ... s} n} and previous cell states (214). The network starts from the initial unit state by using two separate single-layer neural networks applied to the average state of a representative news sequence. and initial hidden state The initialization begins as follows:

[0039]

[0040]

[0041] in, and Where d s The number of neurons in AsyncLSTM is a network parameter trained together with the entire model.

[0042] AsyncLSTM selectively incorporates information learned in the text domain into each step of its inference process on the time series. At each time step l in the sequence (220), AsyncLSTM searches for representative text sequences and assigns probability masses to them, where the assignment is based on alignment with the learned signals in the time series, such as those captured in the last hidden state (224). In one embodiment, the probability mass associated with each news representative vector (212) is computed at the l-th timestamp as follows:

[0043] For l = 1, ..., m

[0044]

[0045] Among them, a l It is an aligned vector, and p l (230) is the probability mass distribution. W a U a ,b a and v a These are the adjustable weights that connect neurons. The weights are defined as follows: as well as Parameter v a Learned to align each vector a l,j The transformation is to a scalar, and the probability quality at the output is ensured through a transfer function (such as a softmax function). The coefficients (230) are the probability quality distributions over the news representative sequence (212). The information from these vectors, scaled by the probability quality of these vectors, is generated by generating an adaptive context vector v as follows. l (240) to influence the learning process on time series:

[0046]

[0047] Here, v0 is initialized as a zero vector. At each time interval t, the final state of coefficient (230) captures the relevant or correlated information of the text modality's performance / behavior relative to the time series modality.

[0048] As shown in the figure, AsyncLSTM constructs the latest context vector as the current representation of the relevant news story, along with the previous context vector v. l-1 The average information between (242). The aggregation vector from the text news modulates the information flow to all gates (including gate0(252), gate1(254), gate2(256), and gate3(258)). The function f used at gate0(252) lIt is the alignment vector between the current state of the time series data and the text news, and is defined as follows: The function i used in gate1(254) l Defined as The function used at gate2(256) Defined as The function o used in gate3(258) l Defined as In each gate, U represents the weight matrix of the neural model, which is learned over time. Then, the current cell state... (260) and hidden state (270) is updated and forwarded as input to the next unit in the AsyncLSTM, where the current state is defined as: And the hidden state is defined as The hidden state is a neural function that infers a new state of time-series data based on time-series input, recurrent neural network (RNN) memory units, and relevant information from the text news. By tightly integrating information learned from the text news domain (e.g., the modality corresponding to the first input feed) into each step of modeling the time-series modality corresponding to the second input feed, the neural network distributes the workload of discovering relevant data in the first input feed through the process of time-series data.

[0049] Relevant data discovered in the first input feed is used to learn patterns in the second input feed. As the AsyncLSTM progresses through the time series, it gradually eliminates data from the first input feed that least matches the patterns learned in the second input feed. The neural network converges to the pattern learned from the last probability mass vector p in the first input feed. m The captured dataset, from which entries are used to explain the significance of each individual news story represented in the first input feed.

[0050] refer to Figure 3A flowchart (300) illustrating the fusion of multimodal data using a recurrent neural network is provided. Two modalities are defined and configured as data feeds, including establishing a first data feed (302) for the first modality and a second data feed (304) for the second modality. The modalities shown and described include text data received in the first data feed and time-series data received in the second data feed. In one embodiment, additional or different modalities may be employed, and therefore, the modalities shown and described herein should not be considered limiting. Each input feed (302) and (304) may have different lengths. The text data in the first data feed is encoded as a vector sequence (306), such as Figure 2 As shown and described. The number of text-encoded vectors at step (306) is based on the sampling size of the text data. The time-series data represented in the second data feed is ingested by AsyncLSTM and encoded into hidden state vectors (308) to learn long-term dependencies. When modeling the time-series modalities, AsyncLSTM performs multiple steps that are interrelated with the text modalities of the first data feed and progressively filters out irrelevant data while retaining data aligned with the current behavioral state and performance of the time-series data (310). More specifically, if relationships exist in the data represented in the feed, the filtering at step (310) includes comparing the time-series data with the text data to determine and perform alignment. The encodings from the first and second data feeds are fused, for example, combined (312), and represent the data alignment from the first data feed, which explains or otherwise describes the behavior of the second data feed during the defined time interval. The fused data (312), for example, the fused vectors, are also referred to herein as inter-modal data. The fused data (312) represents textual data from the first input feed that is closely aligned with the time series behavior and performance from the second input feed. Therefore, the alignment of the data subset across data modalities is determined and returned as output data.

[0051] refer to Figure 4A schematic diagram of a computer system (400) with an integrated artificial intelligence platform supporting the fusion of multimodal data is depicted. As shown, a server (410) is provided that communicates with multiple computing devices (480), (482), (484), (486), (488), and (490) via a network connection (405). The server (410) is configured with a processing unit (412) that communicates with a memory (416) via a bus (414). The server (410) is shown to have an artificial intelligence (AI) platform (450) and is configured to support deep learning neural networks from one or more of the computing devices (480), (482), (484), (486), (488), and (490) via the network (405). More specifically, computing devices (480), (482), (484), (486), (488), and (490) communicate with each other and with other devices or components via one or more wired and / or wireless data communication links, wherein each communication link may include one or more of wires, routers, switches, transmitters, receivers, etc. In this network arrangement, the server (410) and the computer network (405) are capable of communication detection, identification, and resolution. The server (410) communicates effectively with the computer network via communication links (402) and (404). Links (402) and (404) may be wired or wireless. Other embodiments of the server (410) may be used with components, systems, subsystems, and / or devices other than those described herein.

[0052] The AI ​​platform (450) is shown herein as being configured with tools employing deep learning techniques to fuse data across two or more modalities. More specifically, the tool iteratively fuses encoded features from different data modalities in an asynchronous manner. The tool shown herein includes a first data feed manager (452) and a second data feed manager (454). The first data feed manager (452) is operatively coupled to a first data feed. A (452 A The feed A Transmitted via link (402) from one of the operatively coupled devices (480) to (490). In the first data feed... A The data represented in (452A) has a first mode. Similarly, the second data feed manager (454) is operatively coupled to the second data feed. B (452B) This feed B Transmitted via link (402) from one of the operatively coupled devices (480) to (490). In the second data feed... BThe data represented in (452B) has a second modality. The first and second modalities are different. Therefore, the first and second data feed managers (452) and (454) are respectively for heterogeneous data collected from multiple sources via a network connection (405).

[0053] First and second data feeds (452) A ) and (452 B Each of the data in the first data feed manager (452) is processed individually. More specifically, the first data feed manager (452) makes the feed... A (452 A The second data feed manager (454) makes the feed undergo encoding or encoding process. B (452 B ) undergoes encoding or encoding process. First feed (452) A The data is encoded into a first vector set (462) by the first data feed manager (452), and the second feed (452) is encoded into a first vector set (462). B The first and second data feeds, representing different data modalities, are encoded into a second vector set (464) by a second data feed manager (454). In the case of a text-based modality, the corresponding data feed manager learns the semantic dependencies between words and aggregates the corresponding text into a representative vector for each input text document. In one embodiment, the encoding and the corresponding encoding process occur in parallel. Thus, the first and second data feeds, representing different data modalities, are encoded into corresponding vectors.

[0054] As shown in the figure, an artificial recurrent neural network (RNN) analyzer (456) is operatively coupled to the first and second feed managers (452) and (454), respectively. The analyzer is used to model both the encoded first and second data feed modes and to provide an explanation of the behavior of one or both modes. In one embodiment, the RNN is an asynchronous long short-term memory (AsyncLSTM), such as... Figure 1 As shown and described, the analyzer (456) analyzes the first and second encoded datasets (462) and (464), respectively. The analysis includes iteratively and asynchronously fusing encoded features from the first and second data modalities, wherein the fusing includes combining vectors from the first and second datasets, where the combination represents the relevant temporal behavior.

[0055] For example, in one embodiment, the first data feed A (452 A This represents a timestamped text modality, which can take the form of text-based articles, reports, and social media data feeds, and a second data feed. B (452 B) represents the time series data modality. In this example, the analyzer (456) models both the time series modality and the text modality, providing an interpretation of the time series behavior through a concise set of relevant news stories, while maintaining competitive predictive accuracy through iterative and asynchronous fusion. The analysis of the analyzer (456) involves exploring information about the interrelationships between at least two time series from the same or different sampling frequencies. Fusion requires the analyzer (456) to input data from the first data feed (452) A The analyzer (456) filters out one or more representative vectors that are not related to the patterns identified in the encoded second dataset (464). Thus, the analyzer (456) allows and enables the direct incorporation of semantic information learned from the modalities fed from the first data into each time step of behavioral inference of the temporal series data modalities fed from the second data.

[0056] The tools shown herein include, but are not limited to, first and second feed managers (452), (454) and an analyzer (456). The AI ​​platform (450) can receive input from two or more computing devices (480) to (490) via a network (405) and utilize a data source (470) (also referred to herein as a corpus or knowledge base) to selectively access RNNs and other data. As shown, the data source (470) is configured with a library (472) containing multiple datasets and corresponding RNNs created and managed by the analyzer 456. For example, as shown herein, the library (472) is shown to have two RNNs, including RNN0 (474) and RNN1 (476), and associated first and second datasets. RNN0 (474) includes the first dataset (474). A ) and the second dataset (474) B ), where the first and second datasets have different modalities, and RNN1(476) includes the first dataset (476) with different modalities. A ) and the second dataset (476) B First dataset (474) A ) and (476 A The first data feed manager (452) encodes the second dataset (474). B ) and (476 B The dataset is encoded by a second data feed manager (454). In one embodiment, a third dataset including a third data modality may be encoded and managed by an analyzer (456). The number of additional data modalities should not be considered limiting. Thus, the dataset and the corresponding neural model are presented as being local to a knowledge base (470), which is operatively coupled to a server (410) and an AI platform (450).

[0057] It should be understood in this art that the analyzer and the corresponding RNN discover relevant textual information from a first data feed, interpreted in a second data feed, about a given time series. Below is an example of an application of RNNs and their corresponding data exploration. In a financial market use case, the market is predicted to be optimistic or pessimistic. Market predictions are calculated via the second derivative of the adjusted close price of the stock. The first modality is textual data associated with or identifying the stock, and the second modality is time-series data on the stock's price and any price changes over a period of time. Let... and This represents the adjusted transaction price based on three adjacent transaction data points. If... Then the market sentiment on day t+1 is optimistic. If Then the market sentiment for t+1 is pessimistic. If Then the market price on t+1 will rise. If The market price at t+1 will decrease.

[0058] In another example, multimodal data from stock and financial news is used to understand financial market performance, and if signals in the financial news data can be used to understand volatility in the financial markets, and if the next specific segment (e.g., a word or phrase) in the textual news data has future implications for market performance. (Reference) Figure 5 It provides examples of what is being applied in financial markets. Figure 1 A block diagram (500) of the neural network architecture (520) is shown. As shown, the first feed (502) represents a quarterly report, and the second feed (504) represents a news article. Both the quarterly report and the news article are text data and are received by a text module (512) configured for natural language processing (NLP). The text module (512) learns the semantic dependencies between words present in the first feed (502) and the second feed (504) and aggregates them into a representative vector for each document, or in one embodiment, into a representative vector for each news story. The input to the text module (512) at each time period t is a sequence of news stories or articles n. Each news story or news article is analyzed and encoded separately. Similarly, the third data feed (506) is time-series data in the form of stock trading data. The third data feed is received by an operatively coupled asynchronous LSTM (AsyncLSTM) network (530) that learns to generate encoded contextual features. The remaining elements of the neural network architecture are... Figure 1The output (574) from this architecture is shown and described. By highlighting the most relevant news (570) associated with the stock's time-series behavior and performance (572), the output (574) is filtered for news. More specifically, the news (570) and performance behavior (572) are merged into the output (574).

[0059] In various embodiments, the network (405) may include local network connectivity and remote connectivity, enabling the AI ​​platform (450) to operate in environments of any size, including local and global, such as the Internet. Additionally, the AI ​​platform (450) serves as a front-end system that makes various kinds of knowledge extracted from or represented in network-accessible sources and / or structured data sources available. In this way, some processes populate the AI ​​platform (450) with one or more input interfaces or entry points to receive requests and respond accordingly.

[0060] An Application Programming Interface (API) is understood in this art as a software intermediary, such as a calling protocol, between two or more applications that can run on one or more computing environments. Regarding... Figure 4 The AI ​​platform (450) shown and described herein can utilize one or more APIs to support one or more of the tools (452) to (456) and their associated functionalities. References Figure 6 A block diagram (600) is provided illustrating tools (452) to (456) and their associated APIs. As shown, multiple tools are embedded within the AI ​​platform (605), including but not limited to a first data feed manager (452) shown herein and associated with API0 (612), a second data feed manager (454) shown herein and associated with API1 (614), and an analyzer (456) shown herein and associated with API2 (616). Each API supports and enables features such as... Figure 4 The described separately coupled manager functions can be implemented using one or more languages ​​and interface specifications. API0 (612) provides functional support for processing and encoding a first data feed having a first modality; API1 (614) provides functional support for processing and encoding a second data feed having a second modality different from the first modality; and API2 (616) provides functional support for fusing data across two or more modalities and correlating information from at least two time series from the same or different sampling frequencies.

[0061] As shown in the figure, each API (612), (614), and (616) is operatively coupled to an API orchestrator (670) (or orchestration layer), which is understood in the art to serve as an abstraction layer that transparently threads individual APIs together. In one embodiment, the functionality of individual APIs can be combined or combined. Thus, the configuration of the APIs shown herein should not be considered limiting. Therefore, as shown herein, the functionality of the tools can be embodied or supported by their respective APIs.

[0062] The embodiments shown and described herein can take the form of a computer system used with an intelligent computing platform to provide orchestration of activities across one or more domains to minimize risk. Aspects of tools (452) through (456) and their associated functionality can be embodied in a single-location computer system / server, or, in one embodiment, configured in a cloud-based system with shared computing resources. References Figure 7 A block diagram (700) is provided to illustrate an example of a computer system / server (702) (hereinafter referred to as host (702) in a cloud computing environment (710)) to implement the above reference. Figures 1-6 The systems, tools, and processes described. The host (702) operates with a variety of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the host (702) include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and file systems (e.g., distributed storage environments and distributed cloud computing environments) that include any of the aforementioned systems, devices, and their equivalents.

[0063] The host (702) can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The host (702) can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0064] like Figure 7As shown, a host (702) is illustrated in the form of a general-purpose computing device. Components of the host (702) may include, but are not limited to, one or more processors or processing units (704) (e.g., hardware processors), system memory (706), and buses (708) that couple various system components, including system memory (706), to the processors (704). The bus (708) represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of various bus architectures. By way of example and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus. The host (702) typically includes various computer system readable media. This media can be any available media accessible to the host (702), and it includes volatile and non-volatile media, removable and non-removable media.

[0065] The memory (706) may include computer system readable media in the form of volatile memory, such as random access memory (RAM) (730) and / or cache memory (732). By way of example only, the storage system (734) may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown and generally referred to as a "hard disk drive"). Although not shown, a disk drive may be provided for reading from and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media). In this case, each may be connected to a bus (708) via one or more data media interfaces.

[0066] A program / utility 740 having a set (at least one) of program modules 742, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 706, as an example and not a limitation. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules (742) typically perform functions and / or methods of embodiments employing deep learning techniques to fuse data across two or more modalities. For example, the set of program modules (742) may include, for instance, […]. Figure 4 The tools described in (452) to (456).

[0067] The host (702) can also communicate with one or more external devices (714), such as a keyboard, indicating device, etc.; a display (724); one or more devices that enable a user to interact with the host (702); and / or any device that enables the host (702) to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication can occur via one or more input / output (I / O) interfaces (722). Furthermore, the host (702) can communicate with one or more networks (such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet)) via a network adapter (720). As depicted, the network adapter (720) communicates with other components of the host (702) via a bus (708). In one embodiment, multiple nodes of a distributed file system (not shown) communicate with the host (702) via the I / O interface (722) or via the network adapter (720). It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the host (702). Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0068] In this document, the terms “computer program medium,” “computer-usable medium,” and “computer-readable medium” are used to generally refer to media such as main memory (706), which includes RAM (730), cache (732), and storage systems (734), such as removable storage drives and hard disks installed in hard disk drives.

[0069] The computer program (also referred to as computer control logic) is stored in memory (706). The computer program may also be received via a communication interface (such as a network adapter (720)). When run, this computer program enables the computer system to perform features as discussed herein. Specifically, when run, the computer program enables the processing unit (704) to perform features of the computer system. Therefore, this computer program represents the controller of the computer system.

[0070] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0071] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0072] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuits in order to perform aspects of this invention.

[0073] In one embodiment, the host (702) is a node in a cloud computing environment. As is known in the art, cloud computing is a service delivery model for enabling convenient, on-demand network access to a pool of shared configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, VMs, and services) that can be rapidly provisioned and released with minimal management costs or minimal interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models. Examples of these features are as follows:

[0074] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power (such as server time and network storage) on demand without human interaction with the service provider.

[0075] Wide network access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0076] Resource pooling: A provider's computing resources are grouped into resource pools to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Typically, consumers cannot control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center), thus exhibiting location independence.

[0077] Rapid flexibility: Capabilities can be rapidly and flexibly (in some cases automatically) provided to expand outward quickly and be rapidly released to shrink back down. For consumers, the available capacity often appears unlimited and can be purchased at any time and in any quantity.

[0078] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0079] The service model is as follows:

[0080] Software as a Service (SaaS): The capability offered to consumers is the ability to use applications running on a provider's cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities.

[0081] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications on cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0082] Infrastructure as a Service (IaaS): This provides consumers with the capability to deploy and run any software, including operating systems and applications, on the cloud, providing them with processing, storage, networking, and other basic computing resources. Consumers neither manage nor control the underlying cloud infrastructure, but they have control over the operating system, storage, and deployed applications, and may have limited control over chosen network components (e.g., host firewalls).

[0083] The deployment model is as follows:

[0084] Private cloud: A cloud infrastructure that runs exclusively for a single organization. It can be managed by that organization or a third party, and can exist inside or outside the organization.

[0085] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist inside or outside the organization.

[0086] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0087] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursts for load balancing between clouds).

[0088] Cloud computing environments are service-oriented, characterized by statelessness, loose coupling, modularity, and semantic interoperability. The core of computing is the infrastructure comprising a network of interconnected nodes.

[0089] Now for reference Figure 8 The diagram illustrates a cloud computing network (800). As shown, the cloud computing network (800) includes a cloud computing environment (850) with one or more cloud computing nodes (810) that can communicate with local computing devices used by cloud consumers. Examples of these local computing devices include, but are not limited to, personal digital assistants (PDAs) or cellular phones (854A), desktop computers (854B), laptop computers (854C), and / or automotive computer systems (854N). Individual nodes within the nodes (810) can further communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks (such as private clouds, community clouds, public clouds, or hybrid clouds or combinations thereof as described above). This allows the cloud computing environment (800) to provide Infrastructure as a Service, Platform as a Service, and / or Software as a Service without requiring cloud consumers to maintain resources for them on their local computing devices. It should be understood that Figure 8 The various computing devices (854A-N) shown are merely illustrative. The cloud computing environment (850) can communicate with any type of computing device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0090] Now for reference Figure 9 This shows the result of Figure 8 The cloud computing network provides a set of functional abstraction layers (900). First, it should be understood that... Figure 9The components, layers, and functions shown are for illustrative purposes only, and the embodiments are not limited thereto. As depicted, the following layers and corresponding functions are provided: hardware and software layer (910), virtualization layer (920), management layer (930), and workload layer (940).

[0091] The hardware and software layer (910) includes hardware and software components. Examples of hardware components include mainframes, which in one example is... System; a server based on a RISC (Reduced Instruction Set Computer) architecture, in one example being System; IBM System; IBM Systems; storage devices; networks and network components. Examples of software components include network application server software, one example being IBM. Application server software; and database software, in one example being IBM. Database software. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation registered in many jurisdictions worldwide.)

[0092] The virtualization layer (920) provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers; virtual storage; virtual networks, including virtual private networks; virtual applications and operating systems; and virtual clients.

[0093] In one example, the management layer (930) can provide the following functions: resource provisioning, metering and pricing, user portal, service level management, and SLA planning and enforcement. Resource provisioning provides dynamic acquisition of computing resources and other resources used to perform tasks in the cloud computing environment. Metering and pricing provisioning provides cost tracking of resources used within the cloud computing environment and provides bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks and protection for data and other resources. User portal functionality provides consumers and system administrators with access to the cloud computing environment. Service level management functionality provides the allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and enforcement functionality provides pre-scheduling and procurement of cloud computing resources according to the SLA for its projected future needs.

[0094] The workload layer (940) provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include, but are not limited to: mapping and navigation; software development and lifecycle management; virtual classroom teaching provision; data analysis and processing; transaction processing; and deep learning intermodal data analysis.

[0095] It will be understood that this document discloses a system, method, apparatus, and computer program product for evaluating natural language input, detecting queries in corresponding communications, and resolving detected queries with answers and / or supporting content.

[0096] Although specific embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that changes and modifications can be made based on the teachings herein without departing from the invention and its broader aspects. Therefore, the appended claims are intended to cover all such changes and modifications within the true scope of the embodiments. Furthermore, it should be understood that the embodiments are defined solely by the appended claims. Those skilled in the art will understand that if a specific number of claim elements is intentional, such intention will be explicitly stated in the claims, and without such a statement, there is no such limitation. For non-limiting examples, to aid understanding, the appended claims contain the use of the introductory phrases “at least one” and “one or more” to introduce claim elements. However, the use of such phrases should not be construed as implying that introducing a claim element by the indefinite article “a” or “an” would limit any particular claim containing such an introduced claim element to an embodiment containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an”; the same applies to the use of definite articles in the claims.

[0097] Embodiments of the present invention may be systems, methods, and / or computer program products. Furthermore, selected aspects of this embodiment may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and / or hardware aspects, all of which may be collectively referred to herein as “circuit,” “module,” or “system.” Additionally, aspects of embodiments of the present invention may take the form of computer program products implemented in one or more computer-readable storage media having computer-readable program instructions on one or more computer-readable storage media for causing a processor to execute aspects of embodiments of the present invention. The disclosed systems, methods, and / or computer program products thus implemented are operable to improve the functionality and operation of artificial intelligence platforms to employ deep learning techniques to fuse data across two or more modalities.

[0098] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0100] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0102] It should be understood that although specific embodiments have been described herein for illustrative purposes, various modifications may be made without departing from the scope of the embodiments. For example, the data represented in the modalities shown and described herein may include financial data, climate data, medical data, and computer vision data. Therefore, the scope of protection of the embodiments is defined only by the appended claims and their equivalents.

Claims

1. A computer system, comprising: A processor that is operatively coupled to memory; An artificial intelligence (AI) platform, communicating with the processor, has machine learning (ML) tools that employ deep learning techniques to fuse data across modalities, said tools including: A first data feed manager is operatively coupled to a first dataset having a first modality employing a first data format; A second data feed manager is operatively coupled to a second dataset having a second modality employing a second data form, which is different from the first modality; The first data feed manager is configured to encode the first dataset into a first code that includes a first vector set; The second data feed manager is configured to encode the second dataset into a second encoding that includes a second set of vectors; An analyzer, operatively coupled to the first data feed manager and the second data feed manager, is configured to: Explore the correlations between information from at least two time series with the same or different sampling frequencies; This enables the semantic information learned from the first modality of the first dataset to be directly incorporated into each time step of behavioral inference for the second modality of the second dataset; Analyzing encoded first and second datasets using an artificial recurrent neural network (RNN) includes iteratively and asynchronously fusing the first and second encodings, the fusion comprising combining vectors representing relevant temporal behavior from the first and second vector sets; and Returns the fused vector as output data.

2. The system according to claim 1, wherein, The first input data from the first dataset represents a timestamped text data feed, and the second input data from the second dataset represents time series data.

3. The system according to claim 2, wherein, Using an RNN to iteratively and asynchronously fuse the first encoding and the second encoding includes: the RNN being configured to correlate the temporal behavior of the time series data of the second dataset with representative vectors from the first vector set.

4. The system according to claim 2, wherein, Iteratively and asynchronously fusing the first encoding and the second encoding using the RNN includes: the RNN being configured to filter out one or more representative vectors from the first vector set that are unrelated to the pattern determined in the second vector set.

5. The system according to claim 2, wherein, The first data feed manager is configured to encode the first dataset representing the timestamped text data feed by: the first data feed manager being configured to learn semantic dependencies between words in the timestamped text data feed and aggregating the words into a representative vector for each input text document.

6. The system according to claim 1, wherein, The RNN is also configured to explore information from at least two time series with different sampling frequencies and correlate them with each other.

7. A method for fusing data across modalities using deep learning techniques, the method comprising: A computing device receives a multimodal dataset, which includes data from two or more modalities in different formats, including a first dataset having a first modality and a second dataset having a second modality different from the first modality; Processing the first dataset and the second dataset respectively includes: encoding the first dataset into a first encoding including one or more first vectors, and encoding the second dataset into a second encoding including one or more second vectors; Analyzing a processed multimodal dataset includes: exploring the correlations between information from at least two time series with the same or different sampling frequencies; enabling semantic information learned from the first modality of the first dataset to be directly incorporated into each time step of behavioral inference for the second modality of the second dataset; iteratively and asynchronously fusing the first and second encodings, the fusion including combining a first vector and a second vector representing related temporal behaviors; and Returns the fused vector as output data.

8. The method according to claim 7, wherein, The first input data from the first dataset represents a timestamped text data feed, and the second input data from the second dataset represents time series data.

9. The method according to claim 8, wherein, The iterative and asynchronous fusion includes: correlating the temporal behavior of the time series data in the second dataset with a representative first vector.

10. The method according to claim 8, wherein, The iterative and asynchronous fusion includes filtering out one or more representative vectors from the encoded first dataset that are irrelevant to the patterns determined in the encoded second dataset.

11. The method according to claim 8, wherein, Encoding the first dataset representing the timestamped text data feed includes: learning the semantic dependencies between words in the timestamped text data feed, and aggregating the words into a representative vector for each input text document.

12. The method according to claim 7, wherein, The analysis of the processed multimodal dataset further includes: exploring information from at least two time series with different sampling frequencies and correlating them.

13. A method for fusing data across modalities using deep learning techniques, the method comprising: The computing device receives multimodal data, which includes data from two or more modalities in different formats, including at least a first dataset and a second dataset, wherein the first dataset has a first modality and the second dataset has a second modality different from the first modality; Processing the first dataset and the second dataset respectively includes: encoding the first dataset into a first encoding including one or more first vectors, and encoding the second dataset into a second encoding including one or more second vectors; Analyzing processed multimodal data includes: exploring the correlations between information from at least two time series with the same or different sampling frequencies; enabling semantic information learned from the first modality of the first dataset to be directly incorporated into each time step of behavioral inference for the second modality of the second dataset; fusing the first encoding and the second encoding, the fusion including combining a first vector and a second vector representing the correlated temporal behavior between the performance behaviors of data included in the modalities of the multimodal data; and Returns a fused vector that encodes the common behavior.

14. The method according to claim 13, wherein, The first modality is text data, the second modality is time series data, and the fusion further includes: obtaining the attention probability quality of the text data based on the current state of the time series data.

15. The method according to claim 13, wherein, Analyzing the processed multimodal data includes employing deep learning techniques to fuse the data across the modalities.

16. The method of claim 14, wherein, The first input data from the first dataset represents a timestamped text data feed, the second input data from the second dataset represents numerical time series data, and wherein fusing the first vector and the second vector further includes: referring to the numerical time series data in contrast to the timestamped text data.

17. The method according to claim 15, wherein, The fusion of the first vector and the second vector occurs unsupervised.

18. The method according to claim 13, wherein, The multimodal data includes one or more of the following: medical data, climate data, computer vision data, financial data, or a combination thereof.

19. A computer program product for employing deep learning techniques to fuse data across modalities, the computer program product comprising: A computer-readable storage medium, which can be read by processing circuitry, and stores instructions for execution by said processing circuitry to perform the method according to any one of claims 7 to 18.

Images