Interaction control method and interaction control system

Abstract

The application discloses an interactive control method and an interactive control system, and belongs to the technical field of interactive control. The method comprises the following steps: acquiring a current behavior time sequence atlas of a vehicle in a current driving process; in the case that a difference between a current attention distribution of a first large model and a benchmark attention distribution of a current driving scene is less than a preset value, and a duration of the current driving scene exceeds a preset duration, updating the current attention distribution based on the duration; using the first large model with the updated attention distribution, modeling user preferences based on the behavior time sequence atlas, and obtaining user preference information output by the first large model; and determining a target interaction strategy based on the user preference information, and controlling related components in the vehicle to perform an interaction operation based on the target interaction strategy. Thus, the problem of attention collapse of the attention mechanism of dynamic preference modeling in long-term interaction can be effectively avoided, and the adaptability of the interaction strategy is improved.

Description

Technical Field

[0001] This application relates to the field of interactive control technology, specifically to interactive control methods and interactive control systems. Background Technology

[0002] With the rapid development of intelligent connected vehicles, in-vehicle intelligent cockpits have evolved from traditional control centers into intelligent spaces with multimodal interaction capabilities. Achieving personalized and adaptive interaction with users has become a core requirement for improving driving safety and comfort. Related technologies can collect in-vehicle interaction data in real time and perform dynamic preference modeling based on large language models, thus meeting the interaction needs of in-vehicle intelligent cockpits to a certain extent. However, these technologies have significant limitations in long-term interactions, namely the "attention collapse" problem. When high-frequency behaviors (such as minor steering wheel adjustments every 0.5 seconds during long-distance driving) occur repeatedly, the attention mechanism of the large language model in these technologies will over-focus on these high-frequency behaviors, thus ignoring low-frequency but crucial behaviors (such as changes in blink frequency due to fatigue). This makes it impossible for the dynamic preference vector to accurately reflect changes in the user's state (such as changes in music style preferences when fatigued), ultimately leading to a decrease in the adaptability of the interaction strategy (such as failure to switch to soothing music in a timely manner). Summary of the Invention

[0003] This application provides an interactive control method and an interactive control system, which aim to solve the attention collapse problem of the attention mechanism of dynamic preference modeling in long-term interaction, so as to improve the adaptability of interaction strategy.

[0004] Firstly, an interactive control method is provided, including:

[0005] Obtain the current behavior time sequence graph of the vehicle during this driving process. The behavior time sequence graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during this driving process. The edge set includes the connection edges between nodes determined based on the temporal adjacency relationship. The label set includes the driving scenario label of each node.

[0006] If the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds the preset duration, the current attention distribution is updated based on the duration; wherein, the current driving scenario and the duration are determined based on the behavioral time series graph.

[0007] Using the first model updated with attention distribution, user preference modeling is performed based on behavioral time series graphs to obtain user preference information output by the first model.

[0008] The target interaction strategy is determined based on user preference information, and the relevant components in the vehicle are controlled to perform interactive operations based on the target interaction strategy.

[0009] In some of these design approaches, the current attention distribution is updated based on the aforementioned duration, including:

[0010] Based on the aforementioned duration, the redistribution ratio is determined;

[0011] The current attention distribution is updated based on the redistribution ratio and the preset uniform distribution vector of low-frequency events.

[0012] In some of these design approaches, the first major model includes a linear transformation module and a causal convolutional network;

[0013] Using the updated attention distribution model, user preference modeling is performed based on behavioral time-series graphs to obtain user preference information output by the first model, including:

[0014] Using a linear transformation module, at least some of the nodes in the node set are linearly transformed to obtain a first embedding sequence;

[0015] By using a causal convolutional network, the first embedding sequence is processed in the time direction to obtain the second embedding sequence output by the causal convolutional network.

[0016] The second embedding sequence is weighted using the updated current attention distribution to obtain the third embedding sequence;

[0017] User preference modeling is performed based on the third embedding sequence to obtain user preference information.

[0018] In some of these design approaches, the first major model also includes a spiking neural network and a second major model;

[0019] User preference modeling is performed based on the third embedding sequence to obtain user preference information, including:

[0020] For each third embedding representation in the third embedding sequence, the event frequency corresponding to the third embedding representation is determined based on the time interval between the node corresponding to the third embedding representation and the previous node;

[0021] Each third embedding representation with an event frequency greater than a preset frequency is input into a spiking neural network to perform event pulse stream conversion, and the summation result of the event pulse stream output by the spiking neural network is obtained.

[0022] The third embedding representations of events with frequencies less than or equal to a preset frequency are input into the second largest model for context modeling to obtain the semantic representation output by the second largest model.

[0023] The summarized results and semantic representations are fused to generate a fused result.

[0024] Based on the fusion results, user preference information is determined.

[0025] In some of these design approaches, the summarized results and semantic representations are fused to generate a fused result, including:

[0026] Using a pre-defined mapping matrix, the summarized results are converted into feature information with the same dimension as the semantic representation.

[0027] Determine the scene transition penalty coefficient based on the current driving scene label and the previous driving scene label;

[0028] The fusion result is determined based on feature information, semantic representation, and scene transition penalty coefficient.

[0029] In some of these design approaches, the target interaction strategy is determined based on user preference information, including:

[0030] Obtain the vehicle's current original interaction strategy;

[0031] Based on behavioral time series graphs, user preference information, and preset counterfactual conditions, a counterfactual graph is generated, and counterfactual strategies under the counterfactual graph are determined.

[0032] By using the optimization objective function, the target interaction strategy is determined based on the original interaction strategy and the counterfactual strategy.

[0033] In some of these design approaches, counterfactual graphs are generated based on behavioral sequence graphs, user preference information, and preset counterfactual conditions, including:

[0034] Using a conditional variational autoencoder, based on behavioral time series graphs and user preference information, we generate real latent variables and counterfactual latent variables that meet the counterfactual conditions.

[0035] When the distance between the true latent variable and the counterfactual latent variable is greater than a preset distance, the counterfactual latent variable is decoded using a decoder to obtain a counterfactual graph.

[0036] In some of these design approaches, before obtaining the current behavioral timeline of the vehicle during this driving process, the following is also included:

[0037] Acquire multimodal information and vehicle status information collected during the current detection cycle during this driving process;

[0038] The first node is generated based on multimodal information, and the connection edge between the first node and the second node is generated based on the temporal adjacency relationship between the first node and the second node. The driving scenario label of the first node is determined based on vehicle state information. The second node is the previous node of the first node.

[0039] Add the first node to the node set, add the connecting edge between the first node and the second node to the edge set, and add the driving scenario label of the first node to the label set.

[0040] In some of these design approaches, multimodal information includes facial image information, voice information, heart rate information, and steering wheel grip force information;

[0041] The first node is generated based on multimodal information, including:

[0042] Fatigue indicators are determined based on facial image information;

[0043] Generate text embedding vectors based on speech information;

[0044] Extract continuous R intervals from heart rate information and determine heart rate variability based on continuous R intervals;

[0045] Determine the rate of change of grip force based on steering wheel grip force information;

[0046] The aggregated results of fatigue index, text embedding vector, heart rate variability, and grip strength change rate are used as the first node.

[0047] Secondly, an interactive control system is also provided, comprising:

[0048] The acquisition unit is configured to acquire the current behavior time sequence graph of the vehicle during the current driving process. The behavior time sequence graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during the current driving process. The edge set includes the connection edges between nodes determined based on the temporal adjacency relationship. The label set includes the driving scenario label of each node.

[0049] The update unit is configured to update the current attention distribution based on the duration when the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds the preset duration; wherein, the current driving scenario and the duration are determined based on the behavioral time sequence graph.

[0050] The preference modeling unit is configured to use the first large model updated with attention distribution to model user preferences based on behavioral time series graphs and obtain user preference information output by the first large model.

[0051] The control unit is configured to determine a target interaction strategy based on user preference information, and control relevant components in the vehicle to perform interactive operations based on the target interaction strategy.

[0052] Beneficial effects:

[0053] The solution provided in this application embodiment obtains the current behavioral temporal graph of the vehicle during the current driving process. The behavioral temporal graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during the current driving process. The edge set includes connection edges between nodes determined based on temporal adjacency. The label set includes the driving scenario label for each node. Then, if the difference between the current attention distribution of the first large model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds a preset duration, the current attention distribution is updated based on this duration. The current driving scenario and this duration are determined based on the behavioral temporal graph. Next, using the first large model with the updated attention distribution, user preference modeling is performed based on the behavioral temporal graph to obtain the user preference information output by the first large model. Then, a target interaction strategy is determined based on the user preference information, and the relevant components in the vehicle are controlled to perform interactive operations based on the target interaction strategy. By updating the current attention distribution based on the duration when the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds the preset duration, the attention mechanism of dynamic preference modeling can effectively avoid the attention collapse problem in long-term interaction and improve the adaptability of interaction strategy. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of an interactive control method provided in an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the original multimodal data acquisition process provided in the embodiments of this application;

[0057] Figure 3 This is a schematic diagram illustrating the construction process of the behavioral knowledge graph provided in the embodiments of this application;

[0058] Figure 4 This is a schematic diagram of the user preference modeling process provided in an embodiment of this application;

[0059] Figure 5 This is a schematic diagram illustrating the generation process of the target interaction strategy provided in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram of the interactive control system provided in an embodiment of this application.

[0061] Reference numerals: 601-Acquisition unit, 602-Update unit, 603-Preference modeling unit, 604-Control unit. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0064] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0065] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0066] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0067] Related technologies have significant limitations in long-term interactions, namely the "attention collapse" problem. When high-frequency behaviors (such as fine-tuning the steering wheel every 0.5 seconds during long-distance driving) occur repeatedly, the attention mechanism of the large language model in these technologies will over-focus on these high-frequency behaviors, thus ignoring low-frequency but crucial behaviors (such as changes in blink frequency due to fatigue). This makes it impossible for dynamic preference vectors to accurately reflect changes in user state (such as changes in music style needs when fatigued), ultimately leading to a decrease in the adaptability of interaction strategies (such as failure to switch to soothing music in a timely manner).

[0068] This application provides an interactive control method and an interactive control system, which aim to solve the attention collapse problem of the attention mechanism of dynamic preference modeling in long-term interaction, so as to improve the adaptability of interaction strategy.

[0069] In some embodiments, the interactive control method provided in this application is executed by the interactive control system.

[0070] Figure 1 This is a flowchart of an interactive control method provided in an embodiment of this application. For example... Figure 1 As shown, the interactive control method includes the following steps:

[0071] S101: Obtain the current behavior time sequence graph of the vehicle during this driving process. The behavior time sequence graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during this driving process. The edge set includes the connection edges between nodes determined based on the temporal adjacency relationship. The label set includes the driving scenario label of each node.

[0072] S103: If the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds the preset duration, the current attention distribution is updated based on the duration; wherein, the current driving scenario and the duration are determined based on the behavioral time sequence graph.

[0073] S105: Using the first major model updated with attention distribution, user preference modeling is performed based on behavioral time series graph to obtain user preference information output by the first major model;

[0074] S107: Determine the target interaction strategy based on user preference information, and control the relevant components in the vehicle to perform interaction operations based on the target interaction strategy.

[0075] Figure 1 The corresponding embodiment provides a solution that obtains the current behavioral temporal graph of the vehicle during the current driving process. The behavioral temporal graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during the current driving process. The edge set includes connection edges between nodes determined based on temporal adjacency. The label set includes the driving scenario label for each node. Then, if the difference between the current attention distribution of the first large model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds a preset duration, the current attention distribution is updated based on this duration. The current driving scenario and this duration are determined based on the behavioral temporal graph. Next, using the first large model with the updated attention distribution, user preference modeling is performed based on the behavioral temporal graph to obtain the user preference information output by the first large model. Then, a target interaction strategy is determined based on the user preference information, and the relevant components in the vehicle are controlled to perform interactive operations based on the target interaction strategy. By updating the current attention distribution based on the duration when the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds the preset duration, the attention mechanism of dynamic preference modeling can effectively avoid the attention collapse problem in long-term interaction and improve the adaptability of interaction strategy.

[0076] Steps S101 to S107 will be explained below.

[0077] In step S101, the current behavior time sequence map of the vehicle during this driving process is obtained. The behavioral temporal graph G includes a node set V, an edge set E, and a label set C. The node set V is determined based on the user's multimodal information during the current driving process. The edge set E includes connection edges between nodes determined based on temporal adjacency relationships. The label set C includes driving scenario labels for each node. The label set C is based on the vehicle's state information during the current driving process. Confirmed. A single driving scenario label indicates a driving scenario such as commuting, long distance, nighttime, traffic congestion, or highway driving. It should be understood that the classification of driving scenarios in this application can be set according to actual needs and is not specifically limited here. In this application, "user" refers to the vehicle's driver. The driver's multimodal information may include facial image information. Voice information and physiological characteristics information The behavioral temporal graph G serves as the input to the attention-collapse-resistant learning mechanism in the preference modeling stage, laying a contextual graph prior foundation for generating robust personalized interaction strategies.

[0078] Vehicles are equipped with multiple sensor arrays, such as those deployed in the vehicle's smart cockpit. These sensor arrays comprehensively capture the multidimensional behavior and physiological characteristics of the driver in the real-world environment. Once the system is started, without external triggering, these sensor arrays automatically initiate the data acquisition process at a set sampling frequency. These sensor arrays may include a high-resolution driver monitoring camera, a voice recognition module with a microphone, a physiological perception module, and a vehicle status perception module.

[0079] like Figure 2 As shown, the aforementioned multi-sensor array acquires raw multimodal data at a set sampling frequency. This raw multimodal data includes facial image information acquired by a high-resolution driver monitoring camera at a frequency of 30 frames per second (30 frames / second). .in, Figure 2 This is a schematic diagram of the original multimodal data acquisition process provided in the embodiments of this application. Represents the real number space. Facial image information. It can include RGB image sequences. In the 1080×1920×3 format, 1080×1920 represents the resolution of a single frame of facial image, and 3 represents the number of color channels in the facial image. RGB represents the three primary colors: red, green, and blue. This resolution and frame rate enable fine-grained capture of key behavioral features such as facial expressions, eye movements, and head posture, which helps in constructing an accurate node set V.

[0080] The aforementioned raw multimodal data also includes speech information continuously acquired by the speech recognition module at a sampling frequency of 16 kHz (kilohertz). Among them, voice information This includes voice command signals. The 16000 in the text represents a sampling frequency of 16,000 times per second, and 1 represents voice information. The number of channels. This sampling frequency allows the semantics, emotion, and command structure in the speech information to be fully preserved, providing high-quality input for the set of edges E related to semantic interaction in the graph.

[0081] The aforementioned raw multimodal data also includes physiological feature information collected through the physiological sensing module. Physiological characteristics information Including heart rate information ∈ Steering wheel grip information ∈ Its central rate information Steering wheel grip information The sampling frequency can be 10Hz. It should be noted that... Taking the user preference information mentioned above as an example, ,in, Indicates environmental preference, Indicates notification preference, Indicates control preferences. Heart rate information. Steering wheel grip information These two indicators reflect the driver's current level of stress and operational intensity, serving as environmental preferences in subsequent preference modeling. The important physiological basis of component estimation.

[0082] The aforementioned raw multimodal data also includes vehicle state information finely collected by the vehicle state perception module at a sampling frequency of 100Hz. Vehicle status information Including vehicle speed information ∈ Lane offset information ∈ This sampling frequency ensures high temporal resolution capture of vehicle dynamic responses, thereby enabling the accurate construction of a set of context node labels C related to changes in the traffic scene.

[0083] The information from the above-mentioned original multimodal data is aligned and fused along the time axis to form a unified original multimodal input stream: As the core input for subsequent graph construction and preference modeling, it supports the underlying perception capabilities of the entire personalized interaction strategy optimization chain.

[0084] It should be noted that the behavioral timeline graph G of this driving process can be updated in real time. For example, the detection cycle duration can be preset (e.g., 2s, 2.5s, or 3s), where s is the unit, representing seconds. During this driving process, the real-time raw multimodal input stream is acquired in each detection cycle. Based on this original multimodal input stream The system generates nodes, their corresponding connecting edges, and driving scenario labels, and adds this information to the corresponding set in the behavior time sequence graph G. Taking a detection cycle of 2 seconds as an example, each detection cycle can be understood as a 2-second detection window. In this application, as shown... Figure 3 As shown, feature extraction, event aggregation, and structured modeling can be performed on the perceived data within each 2-second detection window, thereby generating graph-structured data with behavioral semantics and driving scenario labels. Figure 3This is a schematic diagram illustrating the construction process of the behavioral knowledge graph G provided in an embodiment of this application.

[0085] As an example, before step S101, the following update steps can be performed on the behavioral knowledge graph G of this driving process:

[0086] Acquire multimodal information of the user and vehicle status information collected during the current detection period during this driving process. ;

[0087] The first node is generated based on the user's multimodal information, and a connection edge between the first and second nodes is generated based on the temporal adjacency relationship between the first and second nodes. Additionally, a connection edge is generated based on vehicle status information. Determine the driving scenario label of the first node; where the second node is the previous node of the first node;

[0088] Add the first node to the node set V, add the connecting edge between the first node and the second node to the edge set E, and add the driving scenario label of the first node to the label set C.

[0089] As described above, a user's multimodal information includes facial image information. Voice information Heart rate information Steering wheel grip information It can specifically obtain the original multimodal input stream during the current driving process within the current detection cycle. .

[0090] Below, with Indicates the first node, Indicates the second node, This represents the edge connecting the first node and the second node. Taking the driving scenario label of the first node as an example, combined with Figure 3 This paper introduces the method for determining the first node, the connecting edge between the first node and the second node, and the driving scene label of the first node.

[0091] Specifically, in generating the first node At this time, the following steps can be performed:

[0092] Based on facial image information Determine fatigue index ;

[0093] Based on voice information Generate text embedding vectors ;

[0094] From heart rate information Extract continuous R intervals and determine heart rate variability based on continuous R intervals. ;

[0095] Based on steering wheel grip force information Determine the rate of change of grip strength ;

[0096] fatigue index Text embedding vectors Heart rate variability and the rate of change of grip strength The aggregation result is used as the first node. .

[0097] Furthermore, based on eyelid state detection algorithms, extract, for example... Figure 3 The PERCLOS (percentage of eyelid closure over the pupil over time) fatigue index shown is as follows:

[0098] .

[0099] Fatigue index Reflecting the driver's visual fatigue level during the window period is an important basis for assessing their attention status.

[0100] voice information Through acoustic modeling and semantic decoding, a text embedding vector of length 50 can be obtained. It is used to capture interactive intent and language behavior features.

[0101] Regarding heart rate information Extract continuous R intervals and calculate heart rate variability in the form of standard deviation. , where n represents the total number of samples with consecutive R intervals. This indicates the time interval between two consecutive heartbeats. The mean RR interval, and HRV (heart rate variability), as an important physiological indicator of sympathetic-parasympathetic balance, participate in modeling environmental preferences in user preference information. .

[0102] Steering wheel grip information The rate of change of grip strength was obtained by calculating the time derivative. This reflects the driver's operational stability and level of tension.

[0103] The above four types of features are concatenated to form the following 54-dimensional node representation vector, which is the first node:

[0104] .

[0105] It should be noted that the first node Specifically, this includes 50-dimensional text embedding vectors. 1D fatigue index 1D heart rate variability 1-dimensional rate of change of grip strength And 1-dimensional global temporal information; this global temporal information includes the first node. The time encoding in the behavioral time sequence map G, or the synchronization check value of the corresponding multimodal features (such as determining whether the current heart rate and grip strength come from the same detection window, 1 for synchronization and 0 for asynchrony).

[0106] In generating the first node Then, the first node Add to node set V. It should be understood that, with a detection period of 2 seconds, features are aggregated every 2 seconds throughout the entire time series, ultimately generating a set of nodes. , where T represents the total number of time segments in the entire driving process.

[0107] The edge set E is constructed based on temporal proximity keys. In the first node... Second node If the absolute value of the time difference between the nodes is less than the preset time difference, then establish the first node. Second node Connecting edges between The preset time difference can be greater than the detection cycle length. Taking a detection cycle length of 2 seconds and a preset time difference of 2.5 seconds as an example, the first node... The timestamp is The second node The timestamp is For example, if the first node Second node The time difference between them satisfies Then establish connecting edges. ,in This is an indicator function; an output of 1 indicates a connection is established, and an output of 0 indicates no connection is established. This is used to generate connection edges. Then, connect the edges. Add to edge set E.

[0108] Regarding vehicle status information It can utilize preset driving scenario recognition strategies based on vehicle status information Determine the first node Driving scenario tags Assuming a pre-defined driving scenario recognition strategy is used, based on vehicle state information... Identifying that the vehicle is currently in a high-speed driving scenario allows for... Among them, "highway" indicates a high-speed route. This is used in generating driving scenario labels. Then, label the driving scenario. Add to tag collection C.

[0109] In step S103, the current attention distribution of the first large model... Baseline attention distribution compared to the current driving scenario The difference between them is less than the preset value, and the duration of the current driving scenario is... If the preset duration is exceeded, the duration will be based on the duration. Update current attention distribution The first major model is an AI (artificial intelligence) model used for user preference modeling. This includes the current driving scenario and its duration. Determined based on the behavioral time sequence graph G. M represents the number of nodes or candidate behavioral events within the current sliding window in the behavioral time sequence graph G. The preset value can be, for example, 0.1 or 0.15, and the preset duration can be, for example, 60 minutes, 61 minutes, or 62 minutes, etc., which can be set according to actual needs and are not specifically limited here.

[0110] As one implementation method, the current attention distribution can be determined using the following formula. Compared with the baseline attention distribution The KL (Kullback-Leibler) divergence between them is used as the degree of difference:

[0111] ;

[0112] in, This represents the KL divergence. , representing the attention intensity of the j-th node. The KL divergence metric measures the degree of difference between the current attention pattern and the long-term stable attention structure. Less than the above preset value and the duration of the current driving scenario If the preset duration is exceeded, it indicates that the first model has been in a stable state for a long time and the attention distribution is changing slowly. At this time, the attention redistribution mechanism is triggered.

[0113] Specifically, it can be based on duration. Determine the redistribution ratio Then based on the redistribution ratio and the preset low-frequency event uniform distribution vector Update the current attention distribution Among them, the uniform distribution vector of low-frequency events This represents the average activation of less-visited nodes in history, encouraging the largest model to actively explore potential but untapped information under redundant attention. Furthermore, the redistribution ratio... The following formula is used to analyze the current attention distribution. Update:

[0114] ;

[0115] in, This represents the updated current attention distribution in the first major model. This update strategy, while maintaining the original attention pattern, introduces subtle perturbations to enhance modeling diversity and ultimately reflect user preference information. It has greater dynamic responsiveness and long-term adaptability, enhancing the accuracy of subsequent counterfactual strategy generation in recognizing fatigue accumulation and low-frequency interaction states.

[0116] In one implementation, the first large model has a target mechanism for updating the attention distribution, which can be as follows: Figure 4 As shown, the behavioral time series graph G is input into the first large model, causing the first large model to utilize the objective mechanism and execute step S103. Among these steps... Figure 4 This is a schematic diagram of the user preference modeling process provided in an embodiment of this application. In one example, the target mechanism can be called an anti-attention collapse mechanism. An anti-attention collapse mechanism is used to address the attention collapse problem that exists in long-term sequences, and dynamically adjusts the attention distribution through mechanisms such as temporal stability gating mechanisms. In one example, the target mechanism includes a temporal stability gating mechanism, which dynamically adjusts the current attention distribution through a control function based on the node state change rate. That is, adjusting the current attention weights of the attention mechanism introduced by the first major model to obtain the updated current attention distribution. Updated current attention distribution The guidance system focuses on time segments where preferences change significantly.

[0117] In step S105, the first master model updated with attention distribution is used to model user preferences based on the behavioral time series graph G, thereby obtaining the user preference information output by the first master model. Among them, user preference information Information in vector form can be called, for example, Figure 4 The user preference vector shown is based on the preceding description of user preference information. ,in, Indicates environmental preference, Indicates notification preference, Indicates control preference, Using the constructed behavioral time series map G as input, the aim is to extract stable and discriminative user preference information from structured multimodal behavioral data.

[0118] Specifically, the first major model includes a linear transformation module and a causal convolutional network; using the updated first major model with attention distribution, user preference modeling is performed based on the behavioral temporal graph G to obtain the user preference information output by the first major model. ,include:

[0119] Using the linear transformation module, at least some nodes in the node set V are linearly transformed to obtain the first embedding sequence. ;

[0120] Using a causal convolutional network, the first embedding sequence is processed in the time direction. The process is performed to obtain the second embedding sequence output by the causal convolutional network. ;

[0121] Utilizing the updated current attention distribution For the second embedded sequence After weighting, the third embedding sequence is obtained. ;

[0122] Based on the third embedding sequence Perform user preference modeling to obtain user preference information .

[0123] It should be noted that the number of at least some of the nodes mentioned above can be related to the updated current attention distribution. The length M is the same. At least some of the nodes mentioned above can be selected from the node set V in ascending order of the difference between the timestamp and the current time.

[0124] Taking a detection period of 2 seconds and a node dimension of 54 as an example, for each node among at least some of the above nodes... ,node This can characterize a multimodal behavior aggregation fragment within 2 seconds. Specifically, the number of at least some of the aforementioned nodes is related to the updated current attention distribution. When the length M is the same, the subscript i here is an integer greater than or equal to 1 and less than or equal to M. In the linear transformation module, for nodes... When performing a linear transformation, it can be mapped to a higher-dimensional latent space representation through a set of linear transformations with shared weights, such as mapping to a 128-dimensional latent space representation, which can be called the first embedding representation. Among them, the weight matrix Bias vector Activation function Enhancing sparsity improves modeling capabilities. After applying a linear transformation to at least some of the nodes using the linear transformation module, the first embedding sequence is obtained. First Embedded Representation Included in the first embedded sequence middle.

[0125] A causal convolutional network can be a three-layer stacked causal convolutional network. The first embedding sequence... This is then processed by a three-layer stacked causal convolutional network, with each convolutional layer oriented temporally to generate the second embedding sequence of the current output. By relying solely on past behavioral information, avoiding information leakage, and preserving the causal structure of sequence modeling, this process enhances the ability to model temporal patterns of behavioral evolution. Among these, the second embedded sequence... Including the first embedded representation The corresponding second embedding representation .

[0126] By utilizing the temporal stability gating mechanism introduced in the first major model, the current attention distribution is dynamically adjusted through a control function based on the node state change rate. Obtain the updated current attention distribution Then, the updated current attention distribution can be utilized. For the second embedded sequence After weighting, the third embedding sequence is obtained. Then based on the third embedding sequence Perform user preference modeling to obtain user preference information .

[0127] Furthermore, the first major model can also include a spiking neural network and a second major model; based on a third embedding sequence. Perform user preference modeling to obtain user preference information It can include:

[0128] For the third embedded sequence For each third embedding representation, the event frequency corresponding to the third embedding representation is determined based on the time interval between the node corresponding to the third embedding representation and the previous node;

[0129] The third embedding representations of events with frequencies greater than a preset frequency are input into a spiking neural network to perform event pulse stream conversion, and the summarized result of the event pulse stream output by the spiking neural network is obtained. ;

[0130] The third embedding representations of events with frequencies less than or equal to a preset frequency are input into the second large model for context modeling to obtain the semantic representation output by the second large model. ;

[0131] Summary results and semantic representation Perform fusion processing to generate fusion result y;

[0132] Based on the fusion result y, determine user preference information. .

[0133] It should be noted that the spiking neural network mentioned above can be called the spiking neural network channel, and the second large model can be called the large model channel. Further, the large model channel can be called the backbone large model channel. To improve the ability to identify different behavioral frequencies, it is necessary to further construct an event frequency discrimination path, guiding short-term high-frequency mutation events to the spiking neural network channel and guiding long-term stable evolutionary information (low-frequency events) to the large model channel. Subsequently, the output results of these two channels are fused through a pulse gating mechanism, outputting a comprehensive preference representation, i.e., the fused result y.

[0134] Finally, the system generates normalized three-dimensional user preference information through the output mapping layer. ,in To output the weight matrix, the Softmax function incorporates user preference information. The sum of the components is 1. User preference information In The component is used to quantify the fatigue level of driver users, in conjunction with the PERCLOS fatigue index. The temporal evolution trend is highly correlated, providing key psychological state basis for generating personalized interaction strategies.

[0135] The pulse modulation separation process takes the deep representation (third embedding representation) of at least some of the aforementioned nodes and their corresponding event frequencies as input. Its purpose is to dynamically guide information into different processing paths based on the occurrence rate of behavioral events on the time axis, thereby improving the system's ability to model user behavior at different rhythms. Specifically, for the third embedding sequence... Each third embedding representation in, such as the third embedding representation ,by Representing the third embedding The timestamp of the corresponding node, in Taking the timestamp of the previous node as an example, the third embedding representation Corresponding event frequency .

[0136] Taking a detection cycle of 2 seconds as an example, the preset frequency can be 0.5Hz. When the event frequency... At 0.5Hz, meaning more than one behavioral event occurs per second, it is considered a high-frequency behavioral characteristic, requiring the introduction of a neural dynamic mechanism to capture transient reactions and sudden preference signals. At this point, a third embedding representation... The dynamic process of the signal is input into the spiking neural network pathway and modeled by a membrane potential differential equation with a leakage mechanism:

[0137] ;

[0138] Where U represents the neuronal membrane potential. It is the membrane potential decay time constant. This is the synaptic weight vector, corresponding to the projected channel embedded in the current node. When the accumulated membrane potential reaches a threshold... This is considered a trigger, and a pulse is emitted. Otherwise, remain silent. This process enables high-frequency nodes to be converted into a sparse stream of event pulses within a specific time window for subsequent gating mechanisms to make fusion decisions.

[0139] When the event frequency At 0.5Hz, the node is considered to belong to a low-frequency, long-term behavior pattern, which is represented by the third embedding representation. It will be directly input into the main large model channel, where context modeling is performed based on the Transformer structure, and a stable semantic representation is output. This channel excels at capturing long-term dependencies and behavioral trend evolution.

[0140] After the entire processing flow is completed, the system summarizes the excitation signals generated by all spiking neural network paths to obtain the summary result. Summary results It reflects the overall activity level of high-frequency preference events within a time segment, is correlated with transient interest components in user preference information, and preserves the semantic representation of the backbone output. This is for subsequent pulse-gated fusion. Here, K represents the number of each third embedded representation input to the spiking neural network channel. This mechanism improves the temporal discriminative ability of preference modeling through frequency-driven information path separation, enabling the system to have stronger dynamic perception and adaptive capabilities when facing complex driving scenarios such as fatigue fluctuations and high-frequency switching in language interaction.

[0141] The pulse-gated fusion mechanism unifies the modeling of pulse dynamics triggered by high-frequency behavioral events with semantic stability in the context of low-frequency behaviors, thereby generating a fusion preference representation vector with temporal responsiveness and contextual consistency. This mechanism accumulates the pulse counts of the high-frequency paths, i.e., the summarizing result. Semantic representation of low-frequency path output For input. Among them, the summary results... and semantic representation Perform fusion processing to generate fusion result y, including:

[0142] Using a pre-defined mapping matrix, the summarized results are... Transformation into semantic representation Feature information with the same dimension ;

[0143] Based on the current driving scenario label and the previous driving scenario label, determine the scenario transition penalty coefficient. ;

[0144] Based on feature information Semantic representation and scene transition penalty coefficient Determine the fusion result y.

[0145] Specifically, feature information .in, It is a pulse signal mapping matrix, responsible for converting quantitative burst responses into continuous characteristics.

[0146] To control the influence ratio of the two types of paths during the fusion process, the system introduces a scene transition penalty coefficient. .in, It can also be called the gating coefficient, which is a weighting factor that is adaptively calculated based on the frequency distribution.

[0147] In one example The calculation formula is:

[0148] ;

[0149] in, The scene embedding vector representing the driving scene label at the current moment. This represents the scene embedding vector of the previous driving scene label (the driving scene label at the previous moment). This calculation formula actually calculates the cosine distance between scene semantics, used to quantify whether the current context is stable. When When below the threshold, such as This indicates that the system has been in the same scene for a long time, at which point it can be... Zeroing is used to suppress the perturbation effect of the pulse path and avoid the model being overly sensitive to repetitive behavior.

[0150] In In the case of resetting to zero, use the zeroed version. Summarize the results Transformation into semantic representation Feature information with the same dimension .

[0151] The fusion result y can then be generated using the following fusion formula:

[0152] ;

[0153] The tanh function is used for output normalization and gradient compression, and is integrated into the formula. Controlling the participation of high-frequency paths, This controls the dominant weights of the low-frequency semantic backbone. This structure makes the model responsive to sudden scene changes and maintains consistent preferences when the scene is stable. The final output is the fusion result. It is then passed to the subsequent preference information decoding module, which uses the formula... Generate user preference information This leads to control preferences Other personalized dimensions possess the ability to jointly perceive frequency and contextual state, effectively supporting the dynamic adjustment of personalized interaction strategies. In one example, the preference information decoding module is a preference vector decoding module, which decodes user preference information. This represents a user preference vector.

[0154] In one implementation, the fusion result y can also be generated using the following fusion formula:

[0155] ;

[0156] in, This represents the Sigmoid function, which enhances the nonlinear expressiveness of the fusion results.

[0157] In step S107, based on user preference information Determine the target interaction strategy And based on the target interaction strategy Control the relevant components in the vehicle to perform interactive operations.

[0158] Furthermore, based on user preference information Determine the target interaction strategy ,include:

[0159] Obtain the vehicle's current original interaction strategy ;

[0160] Based on behavioral time sequence graph G and user preference information Based on preset counterfactual conditions, a counterfactual map is generated. and determining the counterfactual map Counterfactual strategy ;

[0161] By optimizing the objective function, based on the original interaction strategy and counterfactual strategies Determine the target interaction strategy .

[0162] Furthermore, based on behavioral time series graph G and user preference information Based on preset counterfactual conditions, a counterfactual map is generated. ,include:

[0163] Using a conditional variational autoencoder, based on the behavioral temporal graph G and user preference information Generate real latent variables and counterfactual latent variables that meet the counterfactual conditions ;

[0164] In real latent variables Counterfactual hidden variables When the distance between them is greater than a preset distance, the decoder is used to analyze the counterfactual latent variables. Decoding is performed to obtain the counterfactual graph. .

[0165] It should be noted that the optimization objective function is used based on the original interaction strategy. and counterfactual strategies Determined target interaction strategy This can be called a counterfactual enhancement strategy or an enhanced interaction strategy.

[0166] The generation process of counterfactual enhancement strategies uses behavioral time series graph G and user preference information. (like Figure 5 The goal is to construct a set of counterfactual graphs (shown as user preference vectors) that are realistically feasible but whose assumptions change, using the user preference vectors shown as input. .in, Figure 5 This is a schematic diagram illustrating the generation process of the target interaction strategy provided in the embodiments of this application. For example... Figure 5 As shown, firstly, counterfactual conditions are explicitly defined in the processing stage, such as "operational characteristics when the driver is not fatigued." This counterfactual condition semantically guides the generation process to simulate a driving behavior pattern under non-fatigue conditions. Subsequently, based on a conditional variational autoencoder (CVAE), the current behavioral temporal graph G and user preference information are input. The true latent variables are extracted by the conditional variational autoencoder. At the same time, new latent variables are generated under counterfactual conditions, namely counterfactual latent variables. Taking a preset distance of 4.0 as an example, to ensure that the counterfactual samples have sufficient difference, the following constraints are set:

[0167] ;

[0168] Here, the norm represents the Euclidean distance between two latent variables in the latent space; a value greater than 4.0 indicates that the generated sample is semantically significantly different from the original sample. Then, a decoder is used to... Decoding and reconstructing into a counterfactual graph This map shows the sequence of actions a driver might take when not fatigued.

[0169] Next, we introduce the model predictive control (MPC) method to assess the controllability of counterfactual behavior and generate strategies. The objective function is defined as follows:

[0170] ;

[0171] in This represents the control strategy in the original behavior, namely the original interaction strategy mentioned above. In the counterfact map The strategy derived below is the counterfactual strategy mentioned above. and The difference reflects the degree of operational deviation. To optimize the final set of output policies, i.e., the target interaction policy, its norm is used as a sparsity regularization term to suppress over-intervention. The optimized target interaction policy is then output. ,in The optimal control signal sequence, These are node-level preference weights, used to guide subsequent actions based on the counterfactual graph. Training and evaluation of driving decision-making models.

[0172] The interaction optimization execution phase outputs personalized target interaction strategies from the strategy generation phase. As input, the target interaction strategy It is based on user preference information The derived counterfactual reinforcement strategy. When it is identified that the driver user is currently in a state of high cognitive load, such as control preference... At that time, the system will execute a series of perception and interaction adjustment behaviors according to predefined strategies to reduce the user's current perception complexity and operational burden, thereby improving driving safety and user comfort.

[0173] For example, the system will issue control commands to the music playback module, controlling the background music's tempo to be less than 80 beats per minute (BPM) to reduce the intensity of rhythmic stimulation to the auditory system, thereby alleviating the pressure on cognitive resource allocation. Subsequently, for the HUD (head-up display) system, a visibility enhancement strategy will be implemented, increasing the font size of navigation prompts by 20% to improve information acquisition efficiency by increasing text visual salience and reducing multiple gazes and focus adjustments caused by visual blur. Simultaneously, for tactile feedback triggered by human-machine interaction in the cabin, the system will reduce the output amplitude of the seat vibration unit, lowering the current vibration intensity to 50% of its nominal maximum intensity to reduce low-frequency perceptual stimulation from the vibration channel, thus collaboratively achieving a reduction in the overall interactive environment's workload. These operations are issued as commands to the in-vehicle entertainment system and multi-channel actuators through a unified vehicle control interface, ensuring execution synchronization and response consistency, and improving the continuity and intelligence of human-machine interaction response. The processing logic of this stage is related to the control preferences extracted during the user modeling stage. Tightly coupled, and with personalized target interaction strategies output in the strategy generation phase. Using this as the core basis for decision-making, we can achieve rapid deployment and execution of multimodal control schemes.

[0174] In continuous driving scenarios on highways, the system can receive inputs such as duration. The behavioral time-series graph G records key state transitions during driving using a directed graph structure, including operational behavior nodes, physiological signal nodes, and environmental perception nodes, and retains timestamps to form a temporal chain. The processing flow first performs stability analysis on the behavioral time-series graph G, constructing a temporal stability gating mechanism to dynamically weight fatigue-related feature nodes. The function in this gating mechanism calculates the fatigue feature weight adjustment magnitude δ based on the variability of operational behaviors and the fluctuation frequency of physiological indicators in the behavioral time-series graph G. When the following conditions are met... In this way, the influence of relevant fatigue signals such as blink frequency, orientation retention error and breathing rhythm in the graph convolutional unit is enhanced, thereby improving the system's sensitivity to fatigue state.

[0175] Subsequently, addressing the computational load issue caused by frequent steering wheel fine-tuning, the system employs a pulse modulation separation strategy. This strategy identifies steering wheel operations with continuous amplitude changes less than 1° and durations less than 300ms as high-frequency fine-tuning events, performs encoding and compression processing, retaining only frequency statistics without including them in complete feature extraction, effectively reducing tensor dimensionality expansion during map updates. Furthermore, to improve the confidence output... To enhance robustness, the system introduces a counterfactual enhancement mechanism. This mechanism generates a contrast map by constructing feature vectors with slight perturbations, producing a set of positive and negative samples for the same driving segment. During the training phase, a bi-branch adversarial loss function is introduced to enhance the sensitivity of the discrimination boundary to fatigue features. The optimized confidence output makes... The determination is more reliable. The final output is user preference information. Its function is to adjust the personalized human-computer interaction module, when the following conditions are met. At this time, the system maintains a failure rate of less than 3% for detecting fatigue states in continuous scenarios, ensuring the accuracy and safety of interactive adjustments. The duration of the current driving scenario is represented by δ, which represents the weight of fatigue feature enhancement. This is the current confidence level estimate. This information is ultimately used to adjust the user preference settings of the entertainment system and actuators. It should be noted that the δ here and the δ in the previous paragraph are the same parameter as the redistribution ratio δ mentioned earlier.

[0176] Figure 6 This is a schematic diagram of the interactive control system provided in an embodiment of this application. Figure 6 As shown, the interactive control system includes:

[0177] The acquisition unit 601 is configured to acquire the current behavior time sequence graph of the vehicle during the current driving process. The behavior time sequence graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during the current driving process. The edge set includes the connection edges between nodes determined based on the temporal adjacency relationship. The label set includes the driving scene label of each node.

[0178] The update unit 602 is configured to update the current attention distribution based on the duration when the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds the preset duration; wherein the current driving scenario and the duration are determined based on the behavioral time sequence graph.

[0179] Preference modeling unit 603 is configured to use the first large model updated with attention distribution to perform user preference modeling based on behavioral time series graph and obtain user preference information output by the first large model.

[0180] The control unit 604 is configured to determine a target interaction strategy based on user preference information and control relevant components in the vehicle to perform interactive operations based on the target interaction strategy.

[0181] In some embodiments, the update unit 602 is configured to update the current attention distribution based on the aforementioned duration, including:

[0182] Based on the aforementioned duration, the redistribution ratio is determined;

[0183] The current attention distribution is updated based on the redistribution ratio and the preset uniform distribution vector of low-frequency events.

[0184] In some implementations, the first major model includes a linear transformation module and a causal convolutional network;

[0185] Preference modeling unit 603 is configured to utilize the updated first-largest model based on the attention distribution to model user preferences based on the behavioral time series graph, obtaining user preference information output by the first-largest model, including:

[0186] Using a linear transformation module, at least some of the nodes in the node set are linearly transformed to obtain a first embedding sequence;

[0187] By using a causal convolutional network, the first embedding sequence is processed in the time direction to obtain the second embedding sequence output by the causal convolutional network.

[0188] The second embedding sequence is weighted using the updated current attention distribution to obtain the third embedding sequence;

[0189] User preference modeling is performed based on the third embedding sequence to obtain user preference information.

[0190] In some implementations, the first major model also includes a spiking neural network and a second major model;

[0191] Preference modeling unit 603 is configured to perform user preference modeling based on a third embedding sequence to obtain user preference information, including:

[0192] For each third embedding representation in the third embedding sequence, the event frequency corresponding to the third embedding representation is determined based on the time interval between the node corresponding to the third embedding representation and the previous node;

[0193] Each third embedding representation with an event frequency greater than a preset frequency is input into a spiking neural network to perform event pulse stream conversion, and the summation result of the event pulse stream output by the spiking neural network is obtained.

[0194] The third embedding representations of events with frequencies less than or equal to a preset frequency are input into the second largest model for context modeling to obtain the semantic representation output by the second largest model.

[0195] The summarized results and semantic representations are fused to generate a fused result.

[0196] Based on the fusion results, user preference information is determined.

[0197] In some implementations, the preference modeling unit 603 is configured to fuse the summary results and semantic representations to generate a fused result, including:

[0198] Using a pre-defined mapping matrix, the summarized results are converted into feature information with the same dimension as the semantic representation.

[0199] Determine the scene transition penalty coefficient based on the current driving scene label and the previous driving scene label;

[0200] The fusion result is determined based on feature information, semantic representation, and scene transition penalty coefficient.

[0201] In some embodiments, the control unit 604 is configured to determine a target interaction strategy based on user preference information, including:

[0202] Obtain the vehicle's current original interaction strategy;

[0203] Based on behavioral time series graphs, user preference information, and preset counterfactual conditions, a counterfactual graph is generated, and counterfactual strategies under the counterfactual graph are determined.

[0204] By using the optimization objective function, the target interaction strategy is determined based on the original interaction strategy and the counterfactual strategy.

[0205] In some embodiments, the control unit 604 is configured to generate a counterfactual graph based on a behavioral timeline graph, user preference information, and preset counterfactual conditions, including:

[0206] Using a conditional variational autoencoder, based on behavioral time series graphs and user preference information, we generate real latent variables and counterfactual latent variables that meet the counterfactual conditions.

[0207] When the distance between the true latent variable and the counterfactual latent variable is greater than a preset distance, the counterfactual latent variable is decoded using a decoder to obtain a counterfactual graph.

[0208] In some embodiments, the acquisition unit 601 is further configured to:

[0209] Acquire multimodal information and vehicle status information collected during the current detection cycle during this driving process;

[0210] The first node is generated based on multimodal information, and the connection edge between the first node and the second node is generated based on the temporal adjacency relationship between the first node and the second node. The driving scenario label of the first node is determined based on vehicle state information. The second node is the previous node of the first node.

[0211] Add the first node to the node set, add the connecting edge between the first node and the second node to the edge set, and add the driving scenario label of the first node to the label set.

[0212] In some implementations, the multimodal information includes facial image information, voice information, heart rate information, and steering wheel grip force information;

[0213] The acquisition unit 601 is configured to generate a first node based on multimodal information, including:

[0214] Fatigue indicators are determined based on facial image information;

[0215] Generate text embedding vectors based on speech information;

[0216] Extract continuous R intervals from heart rate information and determine heart rate variability based on continuous R intervals;

[0217] Determine the rate of change of grip force based on steering wheel grip force information;

[0218] The aggregated results of fatigue index, text embedding vector, heart rate variability, and grip strength change rate are used as the first node.

[0219] It should be noted that other aspects and implementation details of the interactive control system provided in some embodiments of this application are the same as or similar to the interactive control method described above, and will not be repeated here.

[0220] This application also provides a vehicle including the interactive control system described above.

[0221] This application also provides a computer-readable storage medium storing a computer program thereon, which is loaded by a processor to execute the steps of any of the methods described in the above embodiments. In this application, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0222] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any of the methods described above.

[0223] To address the "attention collapse" problem and enhance the personalization and adaptability of in-vehicle intelligent cockpits in long-term, cross-scenario interactions, this application proposes a personalized interaction optimization method for in-vehicle intelligent cockpits based on a large AI model. This method first collects raw multimodal data from multiple sensor groups within the in-vehicle intelligent cockpit and integrates it into an input stream. Next, it constructs a cross-scenario user behavior time-series graph based on this input stream. Then, it inputs this graph into a large AI model with an anti-attention collapse mechanism to perform multi-scenario dynamic preference modeling, outputting a user preference vector. This vector utilizes linear transformation and causal convolutional networks to process node features, introduces a time-stability gating mechanism to dynamically adjust attention weights, constructs event frequency discrimination paths to process high- and low-frequency events separately, and fuses them using a pulse-gated fusion mechanism. Furthermore, it generates a counterfactual enhancement strategy based on the user preference vector, constructs a counterfactual graph using a conditional variational autoencoder, and optimizes the strategy using a model predictive control method. Finally, it applies the enhanced interaction strategy to the in-vehicle intelligent cockpit control system, driving the cockpit hardware components to execute personalized interactive behaviors.

[0224] This application effectively avoids attention collapse through a temporal stability gating mechanism, enhancing the ability to capture low-frequency key intentions. It improves the modeling accuracy of user behaviors at different rhythms through event frequency discrimination paths and impulse gating fusion mechanisms. Furthermore, the counterfactual enhancement strategy enables the interaction strategy to better adapt to different driving scenarios, especially performing exceptionally well when dealing with low-frequency states such as fatigue accumulation. Simultaneously, meticulous data collection and processing provide accurate and comprehensive underlying data support for subsequent modeling and strategy generation, significantly improving the personalized interactive experience and driving safety of the in-vehicle intelligent cockpit.

[0225] The beneficial effects of this application are derived through reasoning as follows:

[0226] In addressing the "attention collapse" problem, a temporal stability gating mechanism is introduced. Based on the Kullback-Leibler divergence between the current attention distribution and the baseline distribution, attention redistribution is triggered under specific conditions. A uniform distribution vector of low-frequency events is introduced to adjust the weights, which maintains the original attention pattern while actively exploring potential information. This effectively avoids attention collapse and enhances the ability to capture low-frequency key intentions.

[0227] To improve the modeling capabilities for different frequency behaviors, an event frequency discrimination path was constructed. High-frequency behavior events were guided to the spiking neural network channel, and transient features were captured through modeling such as membrane potential differential equations. Low-frequency behavior events were guided to the large model channel to process long-term stable information. The two were then fused through a pulse gating fusion mechanism combined with a scene transition penalty coefficient, thereby improving the modeling accuracy of user behaviors with different rhythms.

[0228] In terms of the adaptability and personalization of the interaction strategy, a counterfactual enhancement strategy is generated based on the user preference vector, a counterfactual graph is constructed through a conditional variational autoencoder, and the strategy is optimized by combining it with model predictive control methods, so that the strategy can cope with more potential scenarios and enhance the adaptability and personalization performance in different driving scenarios.

[0229] In terms of the detail of data acquisition and processing, targeted sampling rates were set for different types of sensors, such as 30 frames / second for cameras and 16kHz for microphones, enabling fine-grained capture of key features. At the same time, multi-dimensional features such as PERCLOS fatigue index and heart rate variability were extracted from the construction of behavioral time series maps, providing more accurate and comprehensive underlying data support for subsequent modeling and strategy generation, and further improving the effectiveness of the overall solution.

[0230] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0231] The interactive control method and interactive control system provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An interactive control method, characterized in that, include: Obtain the current behavior time sequence graph of the vehicle during this driving process. The behavior time sequence graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during this driving process. The edge set includes the connection edges between the nodes determined based on the temporal adjacency relationship. The label set includes the driving scenario label of each node. If the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds a preset duration, the current attention distribution is updated based on the duration; wherein, the current driving scenario and the duration are determined based on the behavioral time sequence graph. Using the first large model updated with attention distribution, user preference modeling is performed based on the behavioral time series graph to obtain the user preference information output by the first large model; Based on the user preference information, a target interaction strategy is determined, and based on the target interaction strategy, the relevant components in the vehicle are controlled to perform interactive operations.

2. The interactive control method according to claim 1, characterized in that, Updating the current attention distribution based on the duration includes: The redistribution ratio is determined based on the duration of the event. The current attention distribution is updated based on the redistribution ratio and the preset uniform distribution vector of low-frequency events.

3. The interactive control method according to claim 1, characterized in that, The first major model includes a linear transformation module and a causal convolutional network; The first large model, updated using the attention distribution, performs user preference modeling based on the behavioral time series graph to obtain user preference information output by the first large model, including: Using the linear transformation module, at least a portion of the nodes in the node set are linearly transformed to obtain a first embedding sequence; The first embedded sequence is processed in the time direction using the causal convolutional network to obtain the second embedded sequence output by the causal convolutional network. The second embedding sequence is weighted using the updated current attention distribution to obtain the third embedding sequence; User preference modeling is performed based on the third embedded sequence to obtain the user preference information.

4. The interactive control method according to claim 3, characterized in that, The first major model also includes a spiking neural network and a second major model; User preference modeling is performed based on the third embedded sequence to obtain the user preference information, including: For each third embedding representation in the third embedding sequence, the event frequency corresponding to the third embedding representation is determined based on the time interval between the node corresponding to the third embedding representation and the previous node; The third embedding representations of events with frequencies greater than a preset frequency are input into the spiking neural network for event pulse stream conversion, and the summary result of the event pulse stream output by the spiking neural network is obtained. The third embedding representations of events with frequencies less than or equal to the preset frequency are input into the second large model for context modeling to obtain the semantic representations output by the second large model. The summarized results and the semantic representations are fused together to generate a fusion result; Based on the fusion results, the user preference information is determined.

5. The interactive control method according to claim 4, characterized in that, The summarized results and the semantic representations are fused to generate a fusion result, including: Using a preset mapping matrix, the summary results are converted into feature information with the same semantic representation dimension; Based on the current driving scenario label and the previous driving scenario label, determine the scenario transition penalty coefficient; The fusion result is determined based on the feature information, the semantic representation, and the scene transition penalty coefficient.

6. The interactive control method according to claim 1, characterized in that, Determining the target interaction strategy based on the user preference information includes: Obtain the vehicle's current original interaction strategy; Based on the behavioral timeline graph, the user preference information, and the preset counterfactual conditions, a counterfactual graph is generated, and a counterfactual strategy under the counterfactual graph is determined; The target interaction strategy is determined by using an optimization objective function, based on the original interaction strategy and the counterfactual strategy.

7. The interactive control method according to claim 6, characterized in that, Based on the behavioral timeline graph, the user preference information, and preset counterfactual conditions, a counterfactual graph is generated, including: Using a conditional variational autoencoder, based on the behavioral time series graph and the user preference information, true latent variables and counterfactual latent variables that meet the counterfactual conditions are generated. If the distance between the true latent variable and the counterfactual latent variable is greater than a preset distance, the counterfactual latent variable is decoded using a decoder to obtain the counterfactual map.

8. The interactive control method according to any one of claims 1-7, characterized in that, Before obtaining the current behavioral time-series graph of the vehicle during this driving process, the method further includes: Acquire the multimodal information and vehicle status information of the vehicle collected during the current detection cycle during this driving process; A first node is generated based on the multimodal information, and a connection edge between the first node and the second node is generated based on the temporal adjacency relationship between the first node and the second node. A driving scenario label for the first node is determined based on the vehicle state information. The second node is the previous node of the first node. Add the first node to the node set, add the connection edge between the first node and the second node to the edge set, and add the driving scenario label of the first node to the label set.

9. The interactive control method according to claim 8, characterized in that, The multimodal information includes facial image information, voice information, heart rate information, and steering wheel grip force information; Generating a first node based on the multimodal information includes: Based on the facial image information, fatigue indicators are determined; Based on the speech information, a text embedding vector is generated; Extract continuous R intervals from the heart rate information, and determine heart rate variability based on the continuous R intervals; Based on the steering wheel grip force information, determine the grip force change rate; The aggregated result of the fatigue index, the text embedding vector, the heart rate variability, and the grip strength change rate is used as the first node.

10. An interactive control system, characterized in that, include: The acquisition unit is configured to acquire the current behavior time sequence graph of the vehicle during the current driving process. The behavior time sequence graph includes a node set, an edge set, and a label set. The node set is determined based on the user's multimodal information during the current driving process. The edge set includes the connection edges between the nodes determined based on the temporal adjacency relationship. The label set includes the driving scenario label of each node. The update unit is configured to update the current attention distribution based on the duration when the difference between the current attention distribution of the first model and the baseline attention distribution of the current driving scenario is less than a preset value, and the duration of the current driving scenario exceeds a preset duration; wherein the current driving scenario and the duration are determined based on the behavior time sequence graph. The preference modeling unit is configured to use the first large model updated with attention distribution to perform user preference modeling based on the behavioral time series graph, and obtain the user preference information output by the first large model. The control unit is configured to determine a target interaction strategy based on the user preference information, and to control relevant components in the vehicle to perform interactive operations based on the target interaction strategy.

Images