A multi-scene multi-modal interaction behavior prediction method and system based on a timing diagram

Abstract

This invention discloses a method and system for predicting multi-scenario, multimodal interactive behavior based on time-series graphs, belonging to the field of human-computer interaction technology for intelligent cockpits. To address the technical problems of difficulty in modeling complex behavioral sequences and significant differences in behavior across multiple scenarios, this invention acquires multimodal user interaction data from multiple scenarios and processes it into event sequences; constructs a time-series graph reflecting temporal transition relationships and encodes it into node representation vectors; uses a multi-task learning model to extract features from the vectors; iteratively trains the model based on the training dataset and the time-series graph structure; and finally uses the trained model to infer the predicted sequences, outputting behavior prediction and scene recognition results. This invention can transform interaction history into explicit graph structure knowledge, overcoming the limitations of traditional linear sequence models, improving the accuracy of long-term interaction prediction, and supporting proactive decision-making in intelligent cockpits.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology in intelligent cockpits, specifically to a method and system for predicting multi-scenario, multi-modal interactive behaviors based on time-series graphs. Background Technology

[0002] With the rapid improvement of the intelligence level of new energy vehicles, the intelligent cockpit has become the core carrier of human-vehicle interaction. The increasing maturity of multimodal interaction methods such as voice control, gesture recognition, and touch operation is gradually replacing traditional physical buttons as the mainstream human-vehicle interaction solution, enabling users to flexibly control in-vehicle functions such as navigation, audio-visual entertainment, and air conditioning while driving. In different driving scenarios, the interaction needs and behavioral habits of cockpit occupants vary. For example, fragmented operations in urban commuting scenarios, efficient and safe operations in highway driving scenarios, and multi-application parallel operations in long-distance travel scenarios. These scenario-based behavioral differences result in cockpit interaction data exhibiting complex temporal correlations, multimodal aliasing, and dynamically changing behavioral patterns, posing challenges to behavior prediction.

[0003] Currently, occupant behavior prediction technologies mainly employ rule-based template matching or end-to-end deep learning methods. Rule-based template matching predicts behavior through predefined causal logic, offering strong interpretability, but it relies heavily on human experience and struggles to adapt to dynamic and diverse driving interaction scenarios. Among deep learning methods, sequence models such as LSTM and Transformer can capture temporal patterns in behavior, but they treat interaction behavior as a linear sequence, making it difficult to handle multi-task, parallel behavior sequences, resulting in lower prediction accuracy for long-term, cross-application interaction behaviors.

[0004] Multi-scenario adaptation is another technical challenge in interactive behavior prediction. Existing technologies mostly employ a single model to cover the entire scenario or train independent models for different scenarios. Single models suffer from performance imbalances due to differences in sample distribution across scenarios; independent models cannot accurately predict cross-scenario boundary behaviors. Some studies use attention mechanisms to improve scenario adaptability, but the models lack the ability to learn scenario-specific patterns. Constructing scenario-based user interaction behavior pattern knowledge is a key approach to improving the accuracy of cross-scenario behavior prediction. Graph Neural Networks (GNNs) are commonly used for modeling entity relationships such as domain knowledge and physical spatial locations, possessing the ability to capture complex topological structures. Gated Graph Neural Networks (GGNNs) introduce gated recurrent units on top of GNNs, updating node features through message passing mechanisms while simultaneously characterizing the topological relationships and temporal evolution patterns between behaviors, solving the problem of difficulty in modeling long-term temporal behavioral dependencies, and providing a technical foundation for accurate modeling of long-term interactive behaviors. Invention patent CN111344717A discloses an interactive behavior prediction method and intelligent device, proposing a multimodal feature extraction and emotion recognition method based on multi-turn dialogue, extracting emotion sequences and predicting interactive behaviors through CNN-LSTM. Invention patent CN120832498A discloses a user behavior prediction system based on multimodal data fusion, proposing a method for multimodal feature extraction and fusion integrating visual, auditory, textual, and physiological signals, and employing Transformer and LSTM for temporal prediction. Invention patent CN121043901A discloses an interactive control method and system, proposing the construction of a behavioral temporal graph containing nodes, temporal adjacency edges, and driving scenario labels based on multimodal information, outputting user preference information through a large model, and optimizing the target interaction strategy based on the counterfactual graph. Invention patent CN120704532A discloses an AI-based adaptive scene intelligent interaction system, proposing the construction of encoders for each type of modality data, employing a layered multi-head attention network to fuse multimodal vectors, and generating user intent representations. Invention patent CN120975228A discloses a collaborative decision-making method for dynamic interaction, proposing the construction of a dynamic interaction flow graph based on multimodal interaction features, and modeling temporal, causal, and logical relationships through graph neural networks, aiming to solve the problem of insufficient dynamic adaptability of traditional robot VLA models. However, none of these existing technologies are based on high-frequency interactive event sequence patterns to build time-series graphs, making it difficult to capture the complex topological relationships between long-term, cross-application, and multi-task interactive behaviors in the in-vehicle cockpit environment; and with only user intent recognition and interaction decision-making as the single objective, they are unable to cope with the differences in user behavior patterns in various scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of existing technologies in capturing the complex topological relationships between multimodal interactive behaviors and in adapting to the differences in user behavior patterns across multiple scenarios. This invention proposes a multi-scenario multimodal interactive behavior prediction method based on time-series graphs. By constructing behavioral time-series graphs and combining them with gated graph neural networks for encoding and inference, this method achieves the beneficial effects of improving the accuracy of long-term cross-application behavior prediction and enabling proactive interactive decision-making in multiple scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A method for predicting multi-scenario, multi-modal interaction behavior based on time sequence graphs includes the following steps: Acquire multimodal user interaction event data in various driving scenarios; The user multimodal interaction event data is preprocessed to obtain a multimodal interaction event sequence; Based on the multimodal interaction event sequence, frequently occurring interaction event sequence patterns are mined, and a multimodal interaction event time sequence diagram is constructed. The nodes in the multimodal interaction event sequence diagram are encoded to obtain node representation vectors; A multi-task learning model is used to extract features from the node representation vector to obtain driving scene recognition features and interaction behavior prediction features. The multimodal interaction event sequence is segmented to construct a training dataset; The multi-task learning model is iteratively trained using the training dataset and the multimodal interaction event time sequence graph. The trained multi-task learning model is used to process the multimodal interaction event sequence to be predicted, and output the interaction behavior prediction results and driving scene recognition results.

[0008] Further, the user multimodal interaction event data is processed to obtain a multimodal interaction event sequence, including: Group user multimodal interaction events by user identifier; Semantically concatenate the attribute values ​​in the grouped user multimodal interaction events to generate semantic-level multimodal interaction events; The semantic-level multimodal interaction events are sorted according to their timestamps to obtain a multimodal interaction event sequence.

[0009] Furthermore, based on the multimodal interaction event sequence, frequently occurring interaction event sequence patterns are mined, and a multimodal interaction event time sequence graph is constructed, including: Extract the recurring subsequences and their frequencies from the multimodal interaction event sequence as high-frequency interaction event sequence patterns; Multimodal interaction events in the high-frequency interaction event sequence pattern are extracted as nodes of the time sequence graph, and directed edges are constructed based on the sequential relationship of adjacent events. The weights of directed edges are determined based on the frequency of occurrence of the high-frequency interaction event sequence pattern, and the weights are normalized to obtain a multimodal interaction event sequence graph.

[0010] Furthermore, the nodes in the multimodal interaction event sequence graph are encoded to obtain node representation vectors, including: Select a node in the multimodal interaction event sequence graph as the starting point, randomly select a successor node to traverse, and generate multiple traversal paths; The node representation vectors in the walking path are optimized and trained based on the co-occurrence probability of each node to obtain the node representation vectors.

[0011] Furthermore, feature extraction is performed on the node representation vector using a multi-task learning model, including: Using the shared encoding module in the multi-task learning model, the node representation vector of the current node is iteratively updated based on the features of neighboring nodes; By using the behavior prediction branch in the multi-task learning model, the updated node representation vector is subjected to nonlinear transformation and attention weighting to obtain interactive behavior prediction features. By aggregating the updated node representation vectors through the scene classification branch in the multi-task learning model, driving scene recognition features are obtained.

[0012] Furthermore, the node representation vector of the current node is iteratively updated based on the characteristics of its neighboring nodes, including: The features of all neighboring nodes of the current node are weighted and aggregated to generate a fused information vector. By using a gating mechanism, effective neighbor information is adaptively filtered between the fused information vector and the current node's state vector to complete the iterative update of the current node's state.

[0013] Further, the multimodal interaction event sequence is segmented to construct a training dataset, including: The multimodal interaction event sequence is segmented according to a preset time interval threshold to obtain multiple continuous segments; For each consecutive segment, the last event is taken as the target event, the events before the last event are taken as the observation sequence, and training samples are constructed by combining the scene category labels.

[0014] Furthermore, the multi-task learning model is iteratively trained using the training dataset and the multimodal interaction event sequence graph, including: For the observation sequences in the training samples, successor nodes are searched from the time sequence graph of multimodal interaction events, and a set of candidate events is determined based on edge weights; An adjacency matrix is ​​constructed based on the observation sequence and the candidate event set, and the confidence scores of the candidate events and the probability distribution of scene recognition are calculated by inputting them into a multi-task learning model. The joint loss of the multi-task model is calculated based on the confidence score and the probability distribution of scene recognition, and the weights of the multi-task learning model are adjusted using an optimizer.

[0015] Further, the multi-task joint loss is calculated based on the confidence score and the probability distribution of scene recognition, including: Calculate the cross-entropy loss between the probability distribution of scene recognition and the scene category label; Calculate the multi-class margin loss between the confidence scores of the target event and the confidence scores of the non-target candidate events; The cross-entropy loss and the multi-class margin loss are weighted and summed to obtain the multi-task joint loss.

[0016] A multi-scenario, multi-modal interaction behavior prediction system based on time sequence graphs, comprising: The data acquisition module is used to acquire user multimodal interaction event data in various driving scenarios; The data preprocessing module is used to preprocess the user multimodal interaction event data to obtain a multimodal interaction event sequence; The sequence diagram construction module is used to construct a multimodal interaction event sequence diagram based on the multimodal interaction event sequence. The node encoding module is used to encode the nodes in the multimodal interaction event sequence diagram to obtain node representation vectors; The feature extraction module is used to extract features from the node representation vector using a multi-task learning model to obtain driving scene recognition features and interaction behavior prediction features. The training data construction module is used to segment the multimodal interaction event sequence into segments and construct a training dataset. The model training module is used to iteratively train the multi-task learning model using the training dataset and the multimodal interaction event time sequence graph. The behavior prediction module is used to process the multimodal interaction event sequence to be predicted using the trained multi-task learning model, and output the interaction behavior prediction results and driving scene recognition results.

[0017] The present invention has achieved the following beneficial effects.

[0018] 1. This invention transforms user interaction history into explicit graph structure knowledge by constructing a time sequence graph, which can characterize the behavior transfer rules and cross-scene associations unique to a scene. It overcomes the technical limitations of traditional linear sequence models in handling multi-task interspersed parallel sequences and improves the model's ability to model complex topological associations.

[0019] 2. The context subgraph extraction method based on probability propagation of this invention effectively reduces the interference of irrelevant nodes on the prediction process by focusing on the local graph structure related to the current observation sequence. Combined with the message passing mechanism of the gated graph neural network, it significantly improves the prediction accuracy of long time series and cross-application interaction behaviors.

[0020] 3. This invention adopts a multi-task learning architecture that includes a shared encoding module, a scene classification branch, and a behavior prediction branch. Through parameter sharing and collaborative training, it realizes joint modeling of driving scene recognition and interactive behavior prediction, enhancing the model's adaptability and generalization ability in different scenarios such as urban commuting and highway driving.

[0021] 4. This invention obtains representation vectors that characterize global topological information by encoding the nodes of a time-series graph, and uses a gated graph neural network for feature extraction and state updates, enabling the model to learn deep semantic connections between behaviors, thus providing reliable technical support for proactive interactive decision-making in intelligent cockpits. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the multi-scenario, multimodal interaction behavior prediction process based on time sequence diagrams in the embodiments. Figure 2 This is a flowchart illustrating the construction of a multimodal interaction event sequence diagram in this embodiment; Figure 3 This is a flowchart illustrating the training dataset construction process in this embodiment; Figure 4 This is a flowchart of the multi-task learning model training process in the embodiment; Figure 5 This is a block diagram of the multi-scenario, multimodal interaction behavior prediction system based on time sequence graphs in the embodiment. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to specific embodiments. These embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operating procedures; however, the scope of protection of the present invention is not limited to the following embodiments.

[0024] This invention provides a method for predicting multi-scenario, multi-modal interaction behavior based on time-series graphs, the processing flow of which is as follows: Figure 1As shown. In this embodiment, the operating platform is a smart cockpit domain controller for new energy vehicles. The domain controller is equipped with a multi-core automotive-grade chip, which supports real-time acquisition of multimodal interactive data and model inference.

[0025] Step S1: Obtain user multimodal interaction event data in various driving scenarios.

[0026] Specifically, the intelligent cockpit domain controller collects user multimodal interaction event datasets in real time. These multimodal interaction events include elements such as user identifier, timestamp, event type, action, control application, operation object, and scene category. The event types include four types: button interaction events, touch interaction events, voice interaction events, and gesture interaction events. The scene categories include urban commuting, highway driving, and long-distance travel.

[0027] In an optional embodiment of the present invention, physical devices such as physical buttons, touch screens, voice assistants, and gesture recognition sensors in the smart cockpit are used to collect interaction data of 20 users over a period of one month. The interaction modes cover four types: button, touch, voice, and gesture. The scenario types cover three core driving scenarios: urban commuting, highway driving, and long-distance travel. The data scale is no less than 50,000 records.

[0028] Step S2: Preprocess the user multimodal interaction event data to obtain a multimodal interaction event sequence.

[0029] Specifically, when preprocessing the user multimodal interaction event data, it involves concatenating attribute values ​​such as event type, action, control application, and operation object to obtain semantic-level multimodal interaction events, such as touching the reading light switch, voice control of the car's air conditioning fan speed, and swiping up to return the human-computer interaction interface to the desktop.

[0030] In an optional embodiment of the present invention, the data subset is divided according to the user identifier, the attribute values ​​of each event are semantically concatenated to generate semantic-level multimodal interaction events such as touching the destination search box of a navigation application or using voice control to adjust the temperature of an air conditioning system, and then sorted in ascending order by timestamp to generate a semantic-level multimodal interaction event sequence for each user.

[0031] In an optional embodiment of the present invention, step S2 may include: Step S21: Group user multimodal interaction events according to user identifier.

[0032] Step S22: Semantically concatenate the attribute values ​​in the grouped user multimodal interaction events to generate semantic-level multimodal interaction events.

[0033] Step S23: Sort the semantic-level multimodal interaction events according to timestamps to obtain a multimodal interaction event sequence.

[0034] Specifically, the sorting process is an ascending sort based on timestamps from smallest to largest.

[0035] Step S3: Based on the multimodal interaction event sequence, mine frequently occurring interaction event sequence patterns and construct a multimodal interaction event time sequence diagram.

[0036] Specifically, the high-frequency interaction event sequence pattern is a frequently occurring subsequence in the multimodal interaction event sequence. The multimodal interaction event time sequence graph is a weighted directed graph, where nodes represent multimodal interaction events, directed edges represent the temporal transition relationships between events, and the weights on the edges represent the probability of transitioning from the current event to the next event.

[0037] In an optional embodiment of the present invention, for any user's semantic-level multimodal interaction event sequence, a high-frequency threshold f=5 is set, and the PrefixSpan algorithm is used to extract subsequences with an occurrence frequency of not less than 5 as high-frequency interaction event sequence patterns. Using multimodal interaction events in the high-frequency patterns as nodes, the sequential transition relationships between events as directed edges, and the occurrence frequency of the pattern as the initial weight of the edges, a multimodal interaction event sequence graph is constructed. The edge weights are normalized so that the sum of the weights of all outgoing edges from the same node is 1.

[0038] In an optional embodiment of the present invention, the construction process of step S3 is as follows: Figure 2 As shown, it may include: Step S31: Extract the recurring subsequences and their frequencies from the multimodal interaction event sequence as high-frequency interaction event sequence patterns.

[0039] Specifically, the PrefixSpan sequence pattern mining algorithm is used to extract subsequences that appear repeatedly and their frequencies from the multimodal interaction event sequence. A high-frequency threshold f is preset, and subsequences with an occurrence frequency of not less than f are extracted as high-frequency interaction event sequence patterns.

[0040] Step S32: Extract the multimodal interaction events in the high-frequency interaction event sequence pattern as nodes of the time sequence graph, and construct directed edges based on the sequential relationship of adjacent events.

[0041] Specifically, the high-frequency interaction event sequence pattern is traversed, and the multimodal interaction events are extracted as nodes of the time sequence graph. Directed edges are constructed based on the sequential relationship between adjacent events, and the frequency of occurrence of the pattern is accumulated as the initial weight of the edge.

[0042] Step S33: Determine the weights of directed edges based on the frequency of occurrence of the high-frequency interaction event sequence pattern, and normalize the weights to obtain a multimodal interaction event time sequence diagram.

[0043] Specifically, the edges in the multimodal interaction event sequence graph are traversed, and the edge weights are normalized so that the sum of the weights of all outgoing edges of the same node is 1, thus obtaining the timing transition probabilities between events.

[0044] Step S4: Encode the nodes in the multimodal interaction event sequence diagram to obtain node representation vectors.

[0045] Specifically, the node representation vector encodes the semantic information of multimodal interaction events in the global topology.

[0046] In an optional embodiment of the present invention, the walk length is set to 10, and a random walk algorithm is used to generate multiple walk paths starting from each node. The generated paths are used as corpus input to a preset training model for training. The context window size is set to 5, the vector dimension is 128, and the number of iteration training rounds is 20, so that nodes with similar topological structures remain in similar positions in the vector space, and finally the representation vector of each node is obtained.

[0047] In an optional embodiment of the present invention, step S4 may include: Step S41: Select a node in the multimodal interaction event sequence diagram as the starting point, randomly select a successor node to traverse, and generate multiple traversal paths.

[0048] Specifically, a random walk algorithm is used, with a preset walk length of [value missing]. Select a node in the multimodal interaction event sequence graph as the starting node, randomly select a successor node of the current node as the next node, and repeat the process until the number of path nodes reaches l, forming a traversal path from the starting node and the path nodes; traverse each node in the multimodal interaction event sequence graph and repeat the above process to generate multiple traversal paths.

[0049] Step S42: Optimize and train the node representation vectors in the walking path based on the co-occurrence probability of each node to obtain the node representation vectors.

[0050] Specifically, each path is treated as a sentence, and the nodes in the path are treated as words. The node representation vectors are initialized using hot-single encoding, and then optimized and trained using the pre-defined training model. The pre-defined training model maximizes the co-occurrence probability of the central node and nodes within its context window, mapping nodes with similar contexts in the graph structure to nearby positions in the vector space. After multiple iterations, the representation vector of each node is obtained. In this embodiment, the pre-defined training model can employ a skip-word model.

[0051] Step S5: Use a multi-task learning model to extract features from the node representation vector to obtain driving scene recognition features and interaction behavior prediction features.

[0052] Specifically, the multi-task learning model comprises three core modules: a shared encoding module, a scene classification branch, and a behavior prediction branch. The two branches share the parameters of the shared encoding module, respectively implementing driving scene recognition and interactive behavior prediction tasks. In this embodiment, the multi-task learning model is built based on a gated graph neural network (GGNN), and the shared encoding module employs a gated graph neural network encoder.

[0053] In an optional embodiment of the present invention, the shared coding module is configured as a 3-layer gated recurrent unit (GRU) with a hidden layer dimension of 128; the behavior prediction branch is configured as a two-layer fully connected network with a hidden layer dimension of 64; and the scene classification branch is configured as an average pooling layer and two fully connected classification layers with an output dimension of 3, corresponding to three types of driving scenarios.

[0054] In an optional embodiment of the present invention, step S5 may include: Step S51: Using the shared encoding module in the multi-task learning model, the node representation vector of the current node is iteratively updated based on the features of the neighboring nodes.

[0055] In an optional embodiment of the present invention, step S51 may include: Step S511: Perform weighted aggregation of the features of all neighboring nodes of the current node to generate a fused information vector.

[0056] Specifically, message passing is performed through the gated recurrent unit in the gated graph neural network, and sparse matrix multiplication is used to weighted aggregate the features of all neighboring nodes of the current node to generate a fused information vector of neighboring nodes.

[0057] Step S512: Use a gating mechanism to adaptively filter effective neighbor information between the fused information vector and the current node's state vector to complete the iterative update of the current node's state.

[0058] Specifically, the aggregated neighbor fusion information vector and the node's current state vector are input into the gated loop unit. The information is adaptively filtered through the gated loop unit's gating mechanism to complete the iterative update of the node's state.

[0059] Step S52: Through the behavior prediction branch in the multi-task learning model, the updated node representation vector is subjected to nonlinear transformation and attention weighting to obtain interactive behavior prediction features.

[0060] Specifically, a two-layer fully connected neural network is used to transform the hidden features of each event node in the observation sequence to generate an attention score. The first layer, a fully connected layer, maps the hidden features of the node to a 64-dimensional feature space and introduces a non-linear transformation using the hyperbolic tangent (Tanh) activation function. The second layer, also a fully connected layer, maps the 64-dimensional features to a 1-dimensional scalar, serving as the attention score for the corresponding event node. Then, a softmax function is used to normalize the attention scores to obtain attention weights. These attention weights are then weighted and summed with the corresponding event feature vectors to obtain the feature representation vector of the observation sequence, which serves as the predicted feature for the interaction behavior. Finally, the negative Euclidean distance between the observation sequence and the candidate events is calculated as the confidence score.

[0061] Step S53: Through the scene classification branch in the multi-task learning model, the updated node representation vector is aggregated to obtain driving scene recognition features.

[0062] Specifically, firstly, pooling layers are used to aggregate the output features of the shared encoding modules of all nodes in the observation sequence to obtain a graph-level feature representation vector, which serves as the driving scene recognition feature. Then, the pooled feature representation vector is input into a fully connected classification network; wherein the fully connected classification network contains several hidden layers and a final output layer. The output layer uses a normalized exponential function to map the feature vector to the probability distribution of each scene category, and the category corresponding to the maximum probability is the scene recognition result.

[0063] Step S6: Segment the multimodal interaction event sequence to construct a training dataset.

[0064] Specifically, the training dataset is used to support the training of a time-series graph-based multi-task learning model.

[0065] In an optional embodiment of the present invention, a time interval threshold is set. =120 seconds, the semantic-level interaction event sequence of each user is segmented into conversation segments. If the difference in timestamps between adjacent events exceeds 120 seconds, they are divided into different segments. For each conversation segment, all events before the last event are taken as the observation sequence, and the last event is taken as the target event. At the same time, the scene category corresponding to the segment is labeled, and training samples containing user ID, observation sequence, target event, and scene category are generated. The training dataset is constructed by traversing all segments and divided into training set, validation set and test set in a ratio of 8:1:1.

[0066] In an optional embodiment of the present invention, the construction process of step S6 is as follows: Figure 3 As shown, it may include: Step S61: The multimodal interaction event sequence is segmented according to a preset time interval threshold to obtain multiple continuous segments.

[0067] Specifically, set a time interval threshold. For each user's semantic-level multimodal interaction event sequence, segmentation is performed. If the time interval between adjacent events is greater than [missing information], [missing information]. This divides the two events into different segments, thus obtaining multiple consecutive segments.

[0068] Step S62: For each consecutive segment, the last event is taken as the target event, the events before the last event are taken as the observation sequence, and training samples are constructed by combining the scene category labels.

[0069] Specifically, for each segment, all events before the last event in the segment are extracted as the observation sequence, the last event is extracted as the target event, and the scene category label to which the session segment belongs is extracted to form a training sample containing user ID, observation sequence, target event, and scene category. By traversing all session segments, the above extraction and construction process is repeated to complete the construction of the training dataset.

[0070] Step S7: Use the training dataset and the multimodal interaction event time sequence diagram to iteratively train the multi-task learning model.

[0071] Specifically, the iterative training process aims to adjust the weight parameters of the multi-task learning model by optimizing the loss function until the model converges.

[0072] In an optional embodiment of the present invention, the Adam optimizer is used to train the model. The multi-task joint loss is used as the optimization objective, with a learning rate of 0.0001, a batch size of 32, and 50 training epochs. After each training epoch, the performance is validated on the validation set, and the weights of the model with the highest recall rate are saved.

[0073] In an optional embodiment of the present invention, the training process of step S7 is as follows: Figure 4 As shown, it may include: Step S71: For the observation sequence in the training samples, search for successor nodes from the multimodal interaction event time sequence graph, and determine the candidate event set based on edge weights.

[0074] Specifically, observation sequences are extracted from the training samples. For the multimodal interaction events within these sequences, successor nodes are searched from the multimodal interaction event time sequence graph to form a set of successor nodes. For each successor node, its joint weight is calculated, where the joint weight is the sum of the edge weights of all events in the observation sequence pointing to that successor node.

[0075] In an optional embodiment of the present invention, the successor nodes are sorted in descending order according to their joint weights, and the nodes with the highest rankings are selected. Each node and the target event in the current sample constitute a candidate event set.

[0076] Step S72: Construct an adjacency matrix based on the observation sequence and the candidate event set, and input it into a multi-task learning model to calculate the confidence score of the candidate events.

[0077] Specifically, nodes and directed edges between nodes are extracted from the observation sequence and candidate event set in the multimodal interaction event time sequence graph to construct an adjacency matrix. The adjacency matrix and the feature vectors of the relevant nodes are then input into the multi-task learning model.

[0078] In an optional embodiment of the present invention, step S72 further includes: normalizing the attention scores of the candidate events using the Softmax function to obtain attention weights; then, weighting and summing the attention weights with the corresponding event feature vectors to obtain the feature representation vector of the observation sequence. For the observation sequence and any candidate event, the negative Euclidean distance between their feature representation vectors is calculated as the confidence score of the candidate event; wherein, the smaller the negative Euclidean distance, the higher the confidence of the candidate event.

[0079] Step S73: Calculate the multi-task joint loss based on the confidence score and the probability distribution of scene recognition, and use an optimizer to adjust the weights of the multi-task learning model.

[0080] Specifically, the Adam optimizer and a multi-task joint loss function are used to optimize the model.

[0081] In an optional embodiment of the present invention, step S73 may include: Step S731: Calculate the cross-entropy loss between the probability distribution of scene recognition and the scene category label.

[0082] Specifically, the formula for calculating the cross-entropy loss is as follows: in, Indicates the number of training samples; Indicates the number of scene categories; Indicates sample Does it belong to the scene category? It belongs to the scene category. The value is 1 if it is not 1, and 0 otherwise. Indicates the model output samples Belongs to the scene category The predicted probability.

[0083] Step S732: Calculate the multi-class margin loss between the confidence score of the target event and the confidence score of the non-target candidate event.

[0084] Specifically, the formula for calculating the multi-class interval loss is as follows: in, This represents the preset classification interval margin value; Indicates sample The target event; This represents the confidence score of the target event; This represents the confidence score of non-target candidate events.

[0085] Step S733: The cross-entropy loss and the multi-class interval loss are weighted and summed to obtain the multi-task joint loss.

[0086] Specifically, the calculation formula for the multi-task joint loss function is as follows: in, This is a combined loss due to multiple tasks; Cross-entropy loss is used for scene classification; For multi-class margin loss used in behavior prediction; and Let these be the weight coefficients of the two loss terms, satisfying... .

[0087] In an optional embodiment of the present invention, the weighting coefficient and Set to 0.4 and 0.6 respectively, these are the marginal values ​​of the classification margin in the multi-class margin loss. Set it to 1.0.

[0088] Step S8: The trained multi-task learning model is used to process the multimodal interaction event sequence to be predicted, and the interaction behavior prediction result and driving scene recognition result are output.

[0089] Specifically, after multiple iterations of training until the loss converges, the parameter weights of the multi-task learning model are fixed, and inference is performed on the real-time input multimodal interaction event sequence.

[0090] In an optional embodiment of the present invention, a trained multi-task learning model is used to score the confidence of candidate events, and the interactive event with the highest confidence is output as the interactive behavior prediction result, while the scene category with the highest probability is output as the driving scene recognition result.

[0091] In an optional embodiment of the present invention, the obtained prediction results are pushed to the intelligent cockpit interactive decision-making system in real time.

[0092] This invention also provides a multi-scenario, multi-modal interaction behavior prediction system based on time-series graphs, such as... Figure 5 As shown, it includes: The data acquisition module is used to acquire user multimodal interaction event data in various driving scenarios; The data preprocessing module is used to preprocess the user multimodal interaction event data to obtain a multimodal interaction event sequence; The sequence diagram construction module is used to mine frequently occurring interaction event sequence patterns based on the multimodal interaction event sequence and construct a multimodal interaction event sequence diagram. The node encoding module is used to encode the nodes in the multimodal interaction event sequence diagram to obtain node representation vectors; The feature extraction module is used to extract features from the node representation vector using a multi-task learning model to obtain driving scene recognition features and interaction behavior prediction features. The training data construction module is used to segment the multimodal interaction event sequence into segments and construct a training dataset. The model training module is used to iteratively train the multi-task learning model using the training dataset and the multimodal interaction event time sequence graph. The behavior prediction module is used to process the multimodal interaction event sequence to be predicted using the trained multi-task learning model, and output the interaction behavior prediction results and driving scene recognition results.

[0093] For interactive behavior prediction tasks, this invention conducts performance tests based on the aforementioned user multimodal interaction event data, selecting the UniSRec model as a baseline for comparison. The UniSRec model uses a Transformer Encoder architecture to model user interaction sequences and is a representative sequence analysis model in the field. This invention uses hit rate (Hit) as a performance evaluation metric. Hit@5 and Hit@10 represent the probability that the model's prediction results hit the actual user interaction events given 5 and 10 interactive behavior prediction candidates, respectively. The comparison results show that this method achieves 92.74% on both Hit@5 and Hit@10 metrics, significantly outperforming the baseline method's 68.36% and 75.28%, respectively, fully demonstrating the effectiveness and advancement of this method in interactive behavior prediction tasks.

[0094] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for predicting multi-scenario, multi-modal interaction behavior based on time-series graphs, characterized in that, Includes the following steps: Acquire multimodal user interaction event data in various driving scenarios; The user multimodal interaction event data is preprocessed to obtain a multimodal interaction event sequence; Based on the multimodal interaction event sequence, frequently occurring interaction event sequence patterns are mined, and a multimodal interaction event time sequence diagram is constructed. The nodes in the multimodal interaction event sequence diagram are encoded to obtain node representation vectors; A multi-task learning model is used to extract features from the node representation vector to obtain driving scene recognition features and interaction behavior prediction features. The multimodal interaction event sequence is segmented to construct a training dataset; The multi-task learning model is iteratively trained using the training dataset and the multimodal interaction event time sequence graph. The trained multi-task learning model is used to process the multimodal interaction event sequence to be predicted, and output the interaction behavior prediction results and driving scene recognition results.

2. The method as described in claim 1, characterized in that, The user multimodal interaction event data is processed to obtain a multimodal interaction event sequence, including: Group user multimodal interaction events by user identifier; Semantically concatenate the attribute values ​​in the grouped user multimodal interaction events to generate semantic-level multimodal interaction events; The semantic-level multimodal interaction events are sorted according to their timestamps to obtain a multimodal interaction event sequence.

3. The method as described in claim 1, characterized in that, Based on the multimodal interaction event sequence, frequently occurring interaction event sequence patterns are mined, and a multimodal interaction event time sequence graph is constructed, including: Extract the recurring subsequences and their frequencies from the multimodal interaction event sequence as high-frequency interaction event sequence patterns; Multimodal interaction events in the high-frequency interaction event sequence pattern are extracted as nodes of the time sequence graph, and directed edges are constructed based on the sequential relationship of adjacent events. The weights of directed edges are determined based on the frequency of occurrence of the high-frequency interaction event sequence pattern, and the weights are normalized to obtain a multimodal interaction event sequence graph.

4. The method as described in claim 1, characterized in that, The nodes in the multimodal interaction event sequence graph are encoded to obtain node representation vectors, including: Select a node in the multimodal interaction event sequence graph as the starting point, randomly select a successor node to traverse, and generate multiple traversal paths; The node representation vectors in the walking path are optimized and trained based on the co-occurrence probability of each node to obtain the node representation vectors.

5. The method as described in claim 1, characterized in that, Feature extraction of the node representation vector is performed using a multi-task learning model, including: Using the shared encoding module in the multi-task learning model, the node representation vector of the current node is iteratively updated based on the features of neighboring nodes; By using the behavior prediction branch in the multi-task learning model, the updated node representation vector is subjected to nonlinear transformation and attention weighting to obtain interactive behavior prediction features. By aggregating the updated node representation vectors through the scene classification branch in the multi-task learning model, driving scene recognition features are obtained.

6. The method as described in claim 5, characterized in that, The node representation vector of the current node is iteratively updated based on the characteristics of its neighboring nodes, including: The features of all neighboring nodes of the current node are weighted and aggregated to generate a fused information vector. By using a gating mechanism, effective neighbor information is adaptively filtered between the fused information vector and the current node's state vector to complete the iterative update of the current node's state.

7. The method as described in claim 1, characterized in that, The multimodal interaction event sequence is segmented to construct a training dataset, including: The multimodal interaction event sequence is segmented according to a preset time interval threshold to obtain multiple continuous segments; For each consecutive segment, the last event is taken as the target event, the events before the last event are taken as the observation sequence, and training samples are constructed by combining the scene category labels.

8. The method as described in claim 1, characterized in that, Using the training dataset and the multimodal interaction event time sequence graph, the multi-task learning model is iteratively trained, including: For the observation sequences in the training samples, successor nodes are searched from the time sequence graph of multimodal interaction events, and a set of candidate events is determined based on edge weights; An adjacency matrix is ​​constructed based on the observation sequence and the candidate event set, and the confidence scores of the candidate events and the probability distribution of scene recognition are calculated by inputting them into a multi-task learning model. The joint loss of the multi-task model is calculated based on the confidence score and the probability distribution of scene recognition, and the weights of the multi-task learning model are adjusted using an optimizer.

9. The method as described in claim 8, characterized in that, The multi-task joint loss is calculated based on the confidence score and the probability distribution of scene recognition, including: Calculate the cross-entropy loss between the probability distribution of scene recognition and the scene category label; Calculate the multi-class margin loss between the confidence scores of the target event and the confidence scores of the non-target candidate events; The cross-entropy loss and the multi-class margin loss are weighted and summed to obtain the multi-task joint loss.

10. A multi-scenario, multi-modal interaction behavior prediction system based on time-series graphs, characterized in that, include: The data acquisition module is used to acquire user multimodal interaction event data in various driving scenarios; The data preprocessing module is used to preprocess the user multimodal interaction event data to obtain a multimodal interaction event sequence; The sequence diagram construction module is used to mine frequently occurring interaction event sequence patterns based on the multimodal interaction event sequence and construct a multimodal interaction event sequence diagram. The node encoding module is used to encode the nodes in the multimodal interaction event sequence diagram to obtain node representation vectors; The feature extraction module is used to extract features from the node representation vector using a multi-task learning model to obtain driving scene recognition features and interaction behavior prediction features. The training data construction module is used to segment the multimodal interaction event sequence into segments and construct a training dataset. The model training module is used to iteratively train the multi-task learning model using the training dataset and the multimodal interaction event time sequence graph. The behavior prediction module is used to process the multimodal interaction event sequence to be predicted using the trained multi-task learning model, and output the interaction behavior prediction results and driving scene recognition results.

Images