Vehicle-mounted intelligent cabin multi-mode interaction control method and system

Abstract

The invention relates to the technical field of man-machine interaction, and discloses a vehicle-mounted intelligent cabin multi-mode interaction control method and system, and the method comprises the steps: collecting a multi-mode interaction signal generated by a user in a vehicle-mounted intelligent cabin, carrying out the atomization decomposition of the multi-mode interaction signal, obtaining atomic-scale modal data, and carrying out the atomization decomposition of the atomic-scale modal data; analyzing an interaction intention corresponding to the atomic-scale modal data; calculating an intention coupling coefficient between the interaction intentions to construct an interaction fusion weight matrix between the interaction intentions; calculating an identification confidence coefficient corresponding to the modal interaction scene, and determining an interaction priority of the interaction intention in the modal interaction scene; a switching delay factor corresponding to the vehicle-mounted intelligent cabin is analyzed; and generating a multi-mode interaction control instruction of the vehicle-mounted intelligent cabin to execute multi-mode interaction control processing of the user in the vehicle-mounted intelligent cabin to obtain a control result. According to the invention, the multi-mode interaction control accuracy of the vehicle-mounted intelligent cabin can be improved.

Description

Technical Field

[0001] This invention relates to a multi-modal interactive control method and system for an in-vehicle intelligent cockpit, belonging to the field of human-computer interaction technology. Background Technology

[0002] The in-vehicle intelligent cockpit is the core carrier of automotive intelligence, integrating multimodal interaction methods such as voice, touch, gesture, and vision to enable convenient information interaction and function control between users and vehicle systems. It not only improves the ease of operation during driving but also reduces driver distraction to ensure driving safety. Therefore, precise control of multimodal interaction in the in-vehicle intelligent cockpit is particularly important. Currently, multimodal interaction control is mainly achieved through single-modal signal recognition and simple command generation. However, existing multi-modal interaction control methods for in-vehicle intelligent cockpits often ignore the correlation between different modal data when processing multi-modal interaction signals, resulting in significant deviations in the recognition of interaction intentions. Furthermore, they fail to adapt to the current in-vehicle environment and scenario requirements, making the judgment of interaction priorities unreasonable. In addition, there is a lack of effective monitoring and optimization of the response delay during device mode switching. When the switching delay is too high, it can easily cause interaction interruption, thereby reducing the smoothness of multi-modal interaction and user experience. Therefore, a method is needed to improve the accuracy of multi-modal interaction control in in-vehicle intelligent cockpits. Summary of the Invention

[0003] This invention provides a multi-mode interactive control method and system for in-vehicle intelligent cockpits, the main purpose of which is to improve the accuracy of multi-mode interactive control in in-vehicle intelligent cockpits.

[0004] To achieve the above objectives, the present invention provides a multi-modal interactive control method for an in-vehicle intelligent cockpit, comprising: The system collects multimodal interaction signals generated by users in the in-vehicle intelligent cockpit, performs atomization decomposition processing on the multimodal interaction signals to obtain atomic-level modal data, and analyzes the interaction intent corresponding to the atomic-level modal data. Based on the atomic-level modal data, the intent coupling coefficient between the interactive intents is calculated to construct the interaction fusion weight matrix between the interactive intents; Identify the modal interaction scenarios in the current in-vehicle environment of the intelligent cockpit, calculate the identification confidence level corresponding to the modal interaction scenario, and combine the interaction fusion weight matrix to determine the interaction priority of the interaction intention in the modal interaction scenario; Monitor the switching response delay of the device mode switching process in the vehicle-mounted intelligent cockpit, and when the switching response delay is higher than a preset delay threshold, analyze the corresponding switching delay factors of the vehicle-mounted intelligent cockpit. Based on the interaction priority, the interaction intent, and the switching delay factor, multi-mode interaction control commands for the in-vehicle intelligent cockpit are generated to execute the multi-mode interaction control processing of the user in the in-vehicle intelligent cockpit and obtain control results.

[0005] Optionally, the step of performing atomization decomposition processing on the multimodal interaction signal to obtain atomic-level modal data includes: The multimodal interaction signals are decoupled to obtain single-modal stream data; The single-modal stream data is subjected to frame slicing to obtain a time-series frame sequence; The time-series frame sequence is subjected to feature extraction processing to obtain the original feature set; The original feature set is subjected to tensor construction processing to obtain the feature tensor; The feature tensor is subjected to primitive decomposition to obtain modal primitives; The modal primitives are subjected to atomic recombination processing to obtain atomic-level modal data.

[0006] Optionally, the step of performing feature extraction processing on the time-series frame sequence to obtain the original feature set includes: The time-series frame sequence is subjected to multi-scale decomposition to obtain scale feature components; The scale feature components are sparsely encoded to obtain sparse feature vectors; The sparse feature vector is subjected to feature enhancement processing to obtain enhanced sparse features; Topological feature extraction is performed on the enhanced sparse features to obtain topological enhanced features; The topology enhancement features are subjected to feature fusion processing to obtain a fused feature vector; Calculate the feature entropy corresponding to the fused feature vector; Based on the feature entropy, the fused feature vector is filtered to obtain the original feature set.

[0007] Optionally, the analysis of the interaction intent corresponding to the atomic-level modal data includes: Extract the modal data identifiers corresponding to the atomic-level modal data, perform semantic parsing on the modal data identifiers, and obtain the initial modal semantics; The semantic relationships between the initial modal semantics are analyzed, and the initial modal semantics are subjected to intent evolution processing to obtain the intent evolution trajectory; Based on the semantic association, the intent evolution trajectory is clustered to obtain a clustered intent set; Calculate the cluster density corresponding to each intent in the clustering intent set; Based on the cluster density, the core clustering intents in the clustering intent set are extracted; Logical verification is performed on the core clustering intent to obtain the interaction intent corresponding to the atomic-level modality data.

[0008] Optionally, calculating the intent coupling coefficient between the interactive intents based on the atomic-level modal data includes: Based on the interaction intent, the atomic-level modal data is aligned to obtain intent modal data; Principal component analysis is performed on the intent modality data to obtain intent modality components; Construct the intent covariance matrix corresponding to the intent modality components; Based on the intent covariance matrix, calculate the modal contribution corresponding to the interaction intent; Based on the modal contribution, the intent coupling coefficient between the interactive intents is calculated.

[0009] Optionally, calculating the intent coupling coefficient between the interactive intents based on the modal contribution includes: The interaction intent is vectorized to obtain an intent feature vector. Calculate the vector similarity between the intent feature vectors; Based on the vector similarity, determine the intent projection score between the interaction intents; The intent coupling coefficient between the interactive intents is calculated by combining the intent projection score and the modality contribution.

[0010] Optionally, calculating the recognition confidence score corresponding to the modal interaction scenario includes: Extract scene environment data from the in-vehicle intelligent cockpit, clean the scene environment data, and obtain target scene environment data; Extract the scene semantic description corresponding to the target scene environment data, and extract the core scene elements from the scene semantic description; The core scene elements are subjected to label mapping processing to obtain scene labels; Based on the scene labels, calculate the recognition confidence level corresponding to the modal interaction scene.

[0011] Optionally, extracting the core scene elements from the scene semantic description includes: The semantic description of the scene is subjected to structured parsing to obtain semantic feature elements; Calculate the scene core degree corresponding to the semantic feature element, and based on the scene core degree, filter out the key feature elements in the semantic feature element; Analyze the semantic relationships between the key feature elements, and based on the semantic relationships, mine the association and combination patterns between the key feature elements; Based on the associated combination pattern and the key feature elements, the core scene elements in the scene semantic description are determined.

[0012] Optionally, the analysis of the switching delay factors corresponding to the in-vehicle intelligent cockpit includes: Collect system operation status data during mode switching, and identify key performance indicators of the in-vehicle intelligent cockpit during mode switching based on the system operation status data; Extract the abnormal fluctuation features from the key performance indicators, and construct the delay feature vector of the in-vehicle intelligent cockpit during mode switching based on the abnormal fluctuation features; Analyze the resource dependency between the aforementioned delay feature vector and the system resource configuration of the in-vehicle intelligent cockpit; By combining the resource dependency and the latency feature vector, the switching latency factor corresponding to the in-vehicle intelligent cockpit is determined.

[0013] To address the aforementioned problems, the present invention also provides a multi-modal interactive control system for an in-vehicle intelligent cockpit, the system comprising: The intent parsing module is used to collect multimodal interaction signals generated by users in the in-vehicle intelligent cockpit, perform atomization decomposition processing on the multimodal interaction signals to obtain atomic-level modal data, and analyze the interaction intent corresponding to the atomic-level modal data. The fusion matrix construction module is used to calculate the intent coupling coefficient between the interactive intents based on the atomic modal data, so as to construct the interaction fusion weight matrix between the interactive intents; The interaction priority determination module is used to identify the modal interaction scenarios in the current vehicle environment of the in-vehicle intelligent cockpit, calculate the recognition confidence level corresponding to the modal interaction scenario, and combine the interaction fusion weight matrix to determine the interaction priority of the interaction intention in the modal interaction scenario. The delay factor analysis module is used to monitor the switching response delay of the device mode switching process in the vehicle-mounted intelligent cockpit. When the switching response delay is higher than a preset delay threshold, the module analyzes the corresponding switching delay factors of the vehicle-mounted intelligent cockpit. The control command generation module is used to generate multi-mode interactive control commands for the in-vehicle smart cockpit based on the interaction priority, the interaction intent, and the switching delay factor, so as to execute the multi-mode interactive control processing of the user in the in-vehicle smart cockpit and obtain the control result.

[0014] Compared to the problems described in the background art, this invention, by performing atomized decomposition processing on the multimodal interaction signals, can accurately separate the core semantic units in voice, touch, and visual signals, avoiding semantic confusion or interference from misoperations, thereby improving the accuracy of subsequent analysis of the interaction intent. This invention, by calculating the intent coupling coefficient between the interaction intents, can quantify the correlation strength and mutual influence between different intents, providing a data foundation for constructing an interaction fusion weight matrix. Furthermore, this invention, by identifying the modal interaction scenarios in the current in-vehicle environment of the intelligent cockpit, can obtain the current environmental interaction context of the intelligent cockpit, and by calculating the recognition confidence level corresponding to the modal interaction scenarios, can verify the reliability of the scene recognition results. To eliminate low-confidence misjudgments and provide reliable scenario basis for subsequent interaction decisions, this invention monitors the switching response delay during the mode switching process of devices in the in-vehicle smart cockpit. When the switching response delay exceeds a preset delay threshold, it analyzes the corresponding switching delay factors in the in-vehicle smart cockpit, accurately identifying factors affecting the smoothness of interaction. This provides an accurate basis for generating subsequent multi-modal interaction control commands. Finally, based on the interaction priority, the interaction intent, and the switching delay factors, this invention generates multi-modal interaction control commands for the in-vehicle smart cockpit to execute the user's multi-modal interaction control processing in the in-vehicle smart cockpit and obtain control results, thereby improving the accuracy of multi-modal interaction control in the in-vehicle smart cockpit. Therefore, the in-vehicle smart cockpit multi-modal interaction control method and system provided by this invention can improve the accuracy of multi-modal interaction control in in-vehicle smart cockpits. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a multi-modal interactive control method for an in-vehicle intelligent cockpit according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the instruction generation process in a multi-modal interactive control method for an in-vehicle intelligent cockpit provided by the present invention; Figure 3 This is a schematic diagram of a module for implementing a multi-mode interactive control system for an in-vehicle intelligent cockpit, as provided in an embodiment of the present invention.

[0016] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] This application provides a multi-modal interactive control method for an in-vehicle intelligent cockpit. The executing entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the multi-modal interactive control method for an in-vehicle intelligent cockpit can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0019] Reference Figure 1 The diagram shown is a flowchart illustrating a multi-modal interactive control method for an in-vehicle intelligent cockpit according to an embodiment of the present invention. In this embodiment, the multi-modal interactive control method for an in-vehicle intelligent cockpit includes: S1. Collect multimodal interaction signals generated by the user in the in-vehicle intelligent cockpit, perform atomization decomposition processing on the multimodal interaction signals to obtain atomic-level modal data, and analyze the interaction intent corresponding to the atomic-level modal data.

[0020] This invention performs atomized decomposition processing on the multimodal interaction signals, which can accurately separate the core semantic units in voice, touch, and visual signals, avoiding semantic confusion or interference from misoperation, thereby improving the accuracy of subsequent analysis of the interaction intent. The multimodal interaction signals are interaction signals generated by the user in the in-vehicle intelligent cockpit through multiple sensing channels, such as voice signals, gesture signals, and facial expression signals. The atomic-level modal data are the basic modal data units obtained after decomposition, such as phonemes in voice, joint coordinates in gestures, and muscle movement units in facial expressions. Furthermore, the acquisition of the multimodal interaction signals can be achieved through acquisition tools, such as an in-vehicle microphone array for capturing voice signals and locating the direction of the sound source, an in-vehicle camera sensor for capturing visual signals of gestures and expressions, and an in-vehicle inertial measurement unit for recording the motion trajectory of gestures, such as an accelerometer unit capable of acquiring the linear acceleration of gestures in real time.

[0021] As an embodiment of the present invention, the step of performing atomization decomposition processing on the multimodal interaction signal to obtain atomic-level modal data includes: The multimodal interaction signals are decoupled to obtain single-modal stream data; The single-modal stream data is subjected to frame slicing to obtain a time-series frame sequence; The time-series frame sequence is subjected to feature extraction processing to obtain the original feature set; The original feature set is subjected to tensor construction processing to obtain the feature tensor; The feature tensor is subjected to primitive decomposition to obtain modal primitives; The modal primitives are subjected to atomic recombination processing to obtain atomic-level modal data.

[0022] The single-modal stream data refers to continuous data of a single modality obtained after separation, such as pure speech stream or independent gesture stream; the temporal frame sequence is a set of frame units arranged in chronological order, constituting the basic time series for analysis; the original feature set is a set of all extracted feature parameters, including multi-dimensional features such as time domain and frequency domain; the feature tensor is a high-dimensional data structure with modal, temporal, and feature dimensions; and the modal primitive is the most basic and indivisible modal representation unit obtained by decomposition.

[0023] Furthermore, the multimodal interaction signals can be decoupled using independent component analysis to obtain single-modal stream data; the single-modal stream data can be sliced ​​using a fixed-duration windowing method to obtain a time-series frame sequence; the original feature set can be reconstructed using a three-dimensional array to form a third-order feature tensor of mode × time × feature; the feature tensor can be decomposed using non-negative tensor decomposition to decompose the high-dimensional tensor into basic tensors and coefficient matrices, obtaining modal primitives representing the most basic modal components; and dynamic time warping combined with graph attention can be used to further refine the data. The force network performs atomic recombination processing on the modal primitives to obtain atomic-level modal data. For example, in speech modal processing, the spectral primitives obtained by non-negative tensor decomposition can represent different phoneme components, while dynamic time warping can align speech primitive sequences at different rates. The graph attention network can learn the contextual dependencies between primitives, such as recognizing the co-occurrence pattern of the "volume increase" primitive and the "speech speed increase" primitive when expressing an urgent intention. Finally, the weighted fusion generates atomic-level modal data with temporal consistency and semantic integrity, providing accurate underlying data support for subsequent interactive intent analysis.

[0024] Furthermore, as an optional embodiment of the present invention, the feature extraction processing of the time-series frame sequence to obtain the original feature set includes: The time-series frame sequence is subjected to multi-scale decomposition to obtain scale feature components; The scale feature components are sparsely encoded to obtain sparse feature vectors; The sparse feature vector is subjected to feature enhancement processing to obtain enhanced sparse features; Topological feature extraction is performed on the enhanced sparse features to obtain topological enhanced features; The topology enhancement features are subjected to feature fusion processing to obtain a fused feature vector; Calculate the feature entropy corresponding to the fused feature vector; Based on the feature entropy, the fused feature vector is filtered to obtain the original feature set.

[0025] Wherein, the scale feature component is a set of features characterizing the local properties of the original signal at different resolutions; the sparse feature vector is a low-dimensional feature representation with high sparsity and representativeness; the enhanced sparse feature is a feature form with stronger discriminativeness and robustness after enhancement; the topologically enhanced feature is an enhanced feature representation containing topological invariants; the fused feature vector is a unified vector representation that integrates the advantages of various features; and the feature entropy is an information-theoretic index that measures the amount of feature information and discriminative power.

[0026] Furthermore, the time-series frame sequence can be multi-scale decomposed using Empirical Mode Decomposition (EMD) to adaptively decompose the non-stationary signal into multiple intrinsic mode functions (IMFs), obtaining feature components reflecting different time scales. The scale feature components can be sparsely encoded using the K-SVD dictionary learning algorithm, constructing an overcomplete dictionary and solving for sparse coefficients to obtain sparse feature vectors with concise representations. The sparse feature vectors can be enhanced using a multi-head self-attention mechanism combined with adversarial training, improving the saliency of key features through attention weight allocation and adversarial example training, thereby extracting enhanced sparse features from the sparse feature vectors. Topological feature extraction can be performed on the enhanced sparse features using persistent cohomology analysis, calculating the Betti number and persistent barcode in the feature space to capture the topological invariant properties of the features, obtaining topologically enhanced features. Gated feature fusion can also be used. The fusion unit performs feature fusion processing on the topology enhancement features, and adopts a gating mechanism to adaptively adjust the weights of different features to obtain a fusion feature vector with complete information. The feature entropy corresponding to the fusion feature vector is calculated, and the information content of each feature is quantified by the information entropy formula. Based on the feature entropy, the fusion feature vector is filtered by a threshold screening method, retaining feature components with information entropy higher than a preset threshold and eliminating redundant features with low information content. Finally, an original feature set that maintains both discriminability and representativeness is obtained. For example, when processing speech modalities, formant features of different frequency bands are obtained through multi-scale decomposition, primitive features representing phonemes are obtained after sparse coding, key phoneme features are enhanced through an attention mechanism, and the topological structure of the acoustic space is extracted using topological analysis. Finally, the most discriminative acoustic features are selected based on feature entropy to form an original feature set that can be used for intent recognition.

[0027] This invention analyzes the interaction intent corresponding to the atomic-level modal data to gain a deeper understanding of the interaction meaning corresponding to user interaction behavior, avoiding ambiguity or misjudgment in intent recognition during multimodal fusion, thereby improving the interaction accuracy and user experience of the in-vehicle intelligent cockpit. The interaction intent is the behavioral purpose or need expressed by the user through multimodal interaction signals, such as navigation control, entertainment adjustment, vehicle settings, etc.

[0028] As an embodiment of the present invention, the analysis of the interaction intent corresponding to the atomic-level modal data includes: Extract the modal data identifiers corresponding to the atomic-level modal data, perform semantic parsing on the modal data identifiers, and obtain the initial modal semantics; The semantic relationships between the initial modal semantics are analyzed, and the initial modal semantics are subjected to intent evolution processing to obtain the intent evolution trajectory; Based on the semantic association, the intent evolution trajectory is clustered to obtain a clustered intent set; Calculate the cluster density corresponding to each intent in the clustering intent set; Based on the cluster density, the core clustering intents in the clustering intent set are extracted; Logical verification is performed on the core clustering intent to obtain the interaction intent corresponding to the atomic-level modality data.

[0029] The modal data identifier is a distinctive feature marker extracted from atomic-level modal data, such as phoneme markers in speech signals, joint motion markers in gesture data, and muscle activity markers in facial expressions. The initial modal semantics is a preliminary semantic understanding of each modal data obtained through semantic parsing, reflecting the meaning of basic semantic units. The semantic correlation is the logical relationship and degree of interdependence between different modal semantics. The intent evolution trajectory is a path sequence describing the evolution of semantic states over time. The clustered intent set is a set of intent categories with similar characteristics obtained after grouping. The cluster density is a quantitative indicator that measures the density of data point distribution within an intent category. The core clustered intent is the main intent category selected based on the cluster density.

[0030] Furthermore, the modal data identifiers corresponding to the atomic-level modal data can be extracted using the Locality Sensitive Hashing (LSH) method. High-dimensional features are mapped to a low-dimensional identifier space, and hash buckets are created according to modal types such as voice, touch, and vision. The low-dimensional vectors within each bucket are then compressed and encoded to obtain the modal data identifiers. Semantic parsing of the modal data identifiers can be performed using a Transformer-based semantic parsing model to obtain initial modal semantics. Graph neural networks can be used to analyze the semantic relationships between the initial modal semantics, constructing a semantic relationship graph. A gated recurrent unit is then used to perform intent evolution processing on the initial modal semantics, capturing the temporal dynamic features of the intent. A spectral clustering algorithm can be used to cluster the intent evolution trajectories based on the semantic relationships to obtain a clustered intent set. The cluster density corresponding to each intent in the clustered intent set is calculated using a kernel density estimation algorithm, such as the Gaussian kernel density estimation algorithm, to evaluate the tightness of each category. Based on the cluster density, the core clustered intents in the clustered intent set can be extracted using a density peak detection method, and categories with a density higher than a preset threshold are selected as core intents. The core clustered intents can be logically verified using a rule reasoning engine based on a knowledge graph, combining domain knowledge and contextual information to verify the rationality and consistency of the intents, and finally outputting verified interaction intents. For example, when analyzing a user's voice command "navigate home" and detecting fatigue expressions, two intents, "route planning" and "safety warning," are obtained through clustering. After logical verification, the dominant intent is determined to be "plan the safest and most comfortable route home."

[0031] S2. Based on the atomic-level modal data, calculate the intent coupling coefficient between the interactive intents to construct the interaction fusion weight matrix between the interactive intents.

[0032] This invention quantifies the correlation strength and mutual influence between different intentions by calculating the intention coupling coefficient between the interactive intentions, providing a data foundation for constructing an interaction fusion weight matrix. The intention coupling coefficient is a quantitative indicator that measures the degree of correlation between different interactive intentions, reflecting the semantic similarity, temporal dependence and logical consistency between intentions.

[0033] As an embodiment of the present invention, calculating the intent coupling coefficient between the interactive intents based on the atomic-level modal data includes: Based on the interaction intent, the atomic-level modal data is aligned to obtain intent modal data; Principal component analysis is performed on the intent modality data to obtain intent modality components; Construct the intent covariance matrix corresponding to the intent modality components; Based on the intent covariance matrix, calculate the modal contribution corresponding to the interaction intent; Based on the modal contribution, the intent coupling coefficient between the interactive intents is calculated.

[0034] The intent modality data is a multimodal data set after spatiotemporal alignment and feature standardization. The intent modality components are feature vectors with the largest variance extracted through principal component analysis, representing the core patterns of intent expression. The intent covariance matrix is ​​a symmetric matrix describing the linear correlation between each intent modality component. The modality contribution is the proportion of variance of each component in the overall intent expression obtained through eigenvalue decomposition. The intent coupling coefficient is an indicator that quantifies the correlation strength between different interactive intents.

[0035] Furthermore, the atomic-level modal data can be spatiotemporally aligned using a dynamic time warping algorithm to eliminate temporal and sampling rate differences between different modal signals; principal component analysis is performed using the variance maximization principle, retaining principal components with a cumulative contribution rate exceeding 80% as intended modal components; an intended covariance matrix is ​​constructed using the Pearson correlation coefficient method, and a symmetric matrix is ​​formed by calculating the covariance relationship between each pair of components; modal contribution is calculated based on the eigenvalue decomposition algorithm, and the proportion of each eigenvalue in the sum of the total eigenvalues ​​is taken as the contribution of the corresponding component.

[0036] Furthermore, as an optional embodiment of the present invention, calculating the intent coupling coefficient between the interactive intents based on the modal contribution includes: The interaction intent is vectorized to obtain an intent feature vector. Calculate the vector similarity between the intent feature vectors; Based on the vector similarity, determine the intent projection score between the interaction intents; The intent coupling coefficient between the interactive intents is calculated by combining the intent projection score and the modality contribution.

[0037] The intent feature vector is a numerical representation obtained by mapping abstract interactive intents to a high-dimensional space through embedding technology; the vector similarity is a metric that quantifies the relative position and directional relationship of different intent feature vectors in the vector space; and the intent projection score is a standardized association score obtained based on vector similarity and through normalization processing.

[0038] Furthermore, the interactive intent can be vectorized using a pre-trained language model, converting the semantic-level intent into a dense vector with rich semantic information, such as the BERT model; the cosine similarity algorithm is used to calculate the vector similarity between the intent feature vectors, and the directional consistency between intents is evaluated by measuring the cosine value of the angle between the vectors; the vector similarity is standardized based on the maximum-minimum normalization method, and linearly mapped to the [0,1] interval to obtain the intent projection score.

[0039] Furthermore, as another embodiment of the present invention, the intent coupling coefficient between the interactive intents is calculated using the following formula, combining the intent projection score and the modal contribution: Where k represents the intent coupling coefficient between interactive intents, and P represents the intent projection score. This represents the modal contribution of the i-th mode. This represents the average value corresponding to the modal contribution. represents the timing decay coefficient, and i represents the mode sequence number.

[0040] Furthermore, This indicates that the mode with the largest contribution is selected, highlighting the key role of the dominant mode. This represents the overall contribution level of all modes, and the temporal decay coefficient is calculated based on the timestamp interval of the interaction intent.

[0041] This invention constructs an interaction fusion weight matrix between interactive intentions by calculating the intention coupling coefficient between the interactive intentions. This enables the fusion and priority allocation of multiple intentions, improving the interaction accuracy and user experience of the in-vehicle intelligent cockpit. The interaction fusion weight matrix is ​​a weight allocation tool for multi-intention decision-making. Furthermore, based on the intention coupling coefficient, the interaction fusion weight matrix between the interactive intentions can be constructed using matrix functions, such as the zero matrix function.

[0042] S3. Identify the modal interaction scenarios in the current in-vehicle environment of the in-vehicle intelligent cockpit, calculate the identification confidence level corresponding to the modal interaction scenario, and combine the interaction fusion weight matrix to determine the interaction priority of the interaction intention in the modal interaction scenario.

[0043] This invention identifies modal interaction scenarios within the current in-vehicle environment of an intelligent in-vehicle cockpit, thereby obtaining the current environmental interaction context. Furthermore, by calculating the recognition confidence level corresponding to each modal interaction scenario, the reliability of the scenario recognition results can be verified, eliminating false judgments with low confidence levels and providing a reliable scenario basis for subsequent interaction decisions. The modal interaction scenarios are typical interaction scenarios in the in-vehicle environment classified based on multimodal interaction features, such as driving navigation scenarios, entertainment control scenarios, and comfort adjustment scenarios. The recognition confidence level is a quantitative value of the credibility of the modal interaction scenario recognition results. Moreover, the recognition of modal interaction scenarios can be achieved through an environmental perception system, such as the aforementioned in-vehicle camera sensor and other monitoring modules.

[0044] As an embodiment of the present invention, calculating the recognition confidence level corresponding to the modal interaction scenario includes: Extract scene environment data from the in-vehicle intelligent cockpit, clean the scene environment data, and obtain target scene environment data; Extract the scene semantic description corresponding to the target scene environment data, and extract the core scene elements from the scene semantic description; The core scene elements are subjected to label mapping processing to obtain scene labels; Based on the scene labels, calculate the recognition confidence level corresponding to the modal interaction scene.

[0045] The scene environment data refers to data collected in the in-vehicle intelligent cockpit regarding the current environment, such as occupant posture and number, and the current vehicle location. The target scene environment data is usable data after the scene environment data has undergone cleaning operations such as noise reduction and outlier processing. The scene semantic description is a structured scene description formed after semantic parsing of the target scene environment data. The core scene elements are the key semantic components in the scene semantic description that affect scene recognition, including user state, environmental conditions, and interaction intent. The scene tag is a standardized scene identifier obtained by matching the core scene elements with a preset scene type library. The preset scene type library is a structured knowledge base that includes typical interactive scene features such as navigation, entertainment, and communication, and is pre-built through training with a large amount of scene sample data.

[0046] Furthermore, the scene environment data can be cleaned using the isolated forest algorithm to identify and remove abnormal interaction data points, thus obtaining the target scene environment data. Core scene elements can be extracted from the scene semantic description using an attention mechanism compiled from a programming language. The core scene elements can be labeled using a knowledge graph mapping method, matching the elements with nodes in the scene type knowledge base to obtain scene labels. A softmax classifier can be used to calculate the recognition confidence score corresponding to the modal interaction scene based on the scene labels. Specifically, this is done by calculating the matching probability distribution between the current scene description and scene templates of each category, taking the maximum probability value as the recognition confidence score. For example, when recognizing a "navigation scene," the system calculates the matching degree between the current scene elements and each preset scene category. If the matching probability between navigation-related elements and the "navigation scene" template reaches 0.92, this value is output as the recognition confidence score, thereby achieving a reliability assessment of the modal interaction scene recognition results.

[0047] Furthermore, as an optional embodiment of the present invention, the extraction of core scene elements from the scene semantic description includes: The semantic description of the scene is subjected to structured parsing to obtain semantic feature elements; Calculate the scene core degree corresponding to the semantic feature element, and based on the scene core degree, filter out the key feature elements in the semantic feature element; Analyze the semantic relationships between the key feature elements, and based on the semantic relationships, mine the association and combination patterns between the key feature elements; Based on the associated combination pattern and the key feature elements, the core scene elements in the scene semantic description are determined.

[0048] The semantic feature element is the smallest semantic unit obtained after structured parsing of the scene semantic description, including basic semantic information such as entities, actions, attributes, and environment in the scene; the scene core degree is a quantitative indicator that measures the importance of the semantic feature element in expressing the core meaning of the scene, and the higher the value, the stronger the fit of the feature element to the scene description; the key feature element is a high-value feature unit with a scene core degree higher than a preset threshold selected from the semantic feature elements, which is the basis for constituting the core elements of the scene; the semantic association relationship is the semantic-level connection between key feature elements, such as causal relationship, subordinate relationship, and cooperative relationship; the association combination pattern is a stable combination form of key feature elements mined based on semantic association relationships, reflecting the inherent organizational rules of the core elements in the scene.

[0049] Furthermore, the semantic description of the scene can be structured and parsed using a dependency parsing algorithm to identify the subject-verb-object structure and modification relationships in the sentence, thereby obtaining semantic feature elements. The scene core degree corresponding to the semantic feature elements can be calculated using a random forest feature importance evaluation method. Specifically, this is achieved by calculating the contribution weight of each feature element to the classification accuracy in the scene classification task, and selecting the top N feature elements with the highest contribution as key feature elements. Finally, the semantic association relationships between the key feature elements can be analyzed using the Apriori association rule mining algorithm to identify those with support and confidence levels exceeding a preset threshold. Associative combination patterns: Based on the strength index of the associated combination patterns and the scene coreness of key feature elements, a weighted scoring strategy can be used to determine the core scene elements from the key feature elements. For example, for the semantic description of "navigation scene", semantic feature elements such as "voice command - destination query", "gesture - map zoom", and "vehicle status - low speed driving" are obtained after parsing. "Destination query" and "map zoom" are selected as key feature elements by calculating contribution. Further mining reveals that there is a strong association combination pattern between these two feature elements, which is finally determined as the core scene element, fully representing the core semantics of the navigation interaction scene.

[0050] This invention calculates the recognition confidence level corresponding to the modal interaction scenario and combines it with the interaction fusion weight matrix to determine the interaction priority of the interaction intent in the modal interaction scenario. This allows for understanding the importance ranking of different interaction intents in the current scenario, ensuring the adaptability of the interaction response to the in-vehicle scenario and the matching degree of user needs. The interaction priority is a quantitative indicator of the execution order of the interaction intent in the modal interaction scenario. Further, the interaction priority of the interaction intent in the modal interaction scenario is determined by combining the interaction fusion weight matrix and the recognition confidence level. The steps are as follows: First, the interaction fusion weight matrix is ​​weighted and adjusted based on the recognition confidence level to obtain a scenario-adaptive weight matrix; then, the priority score of each interaction intent under the scenario-adaptive weight matrix is ​​calculated; finally, the interaction priority order of the interaction intents is generated by sorting them from high to low according to the priority scores.

[0051] S4. Monitor the switching response delay of the device mode switching process in the vehicle-mounted intelligent cockpit, and when the switching response delay is higher than a preset delay threshold, analyze the corresponding switching delay factors of the vehicle-mounted intelligent cockpit.

[0052] This invention monitors the switching response delay during the mode switching process of devices in an in-vehicle intelligent cockpit. When the switching response delay exceeds a preset delay threshold, it analyzes the corresponding switching delay factors of the in-vehicle intelligent cockpit, accurately identifying factors affecting the smoothness of interaction. This provides an accurate basis for the generation of subsequent multi-modal interactive control commands. The switching response delay is a measure of the time delay from triggering a mode switching command to the complete response of the target mode. The preset delay threshold is the maximum allowable delay set according to the requirements of smooth interaction. The switching delay factors are the internal and external system causes that lead to delays in the mode switching process. Furthermore, the monitoring of the device mode switching process can be achieved using a high-precision timestamp recording method.

[0053] As an embodiment of the present invention, the analysis of the switching delay factors corresponding to the in-vehicle intelligent cockpit includes: Collect system operation status data during mode switching, and identify key performance indicators of the in-vehicle intelligent cockpit during mode switching based on the system operation status data; Extract the abnormal fluctuation features from the key performance indicators, and construct the delay feature vector of the in-vehicle intelligent cockpit during mode switching based on the abnormal fluctuation features; Analyze the resource dependency between the aforementioned delay feature vector and the system resource configuration of the in-vehicle intelligent cockpit; By combining the resource dependency and the latency feature vector, the switching latency factor corresponding to the in-vehicle intelligent cockpit is determined.

[0054] The system operation status data refers to time-series data collected from the cockpit operating system kernel and hardware sensors before and after the mode switching event is triggered, including but not limited to CPU utilization, memory bandwidth, I / O interrupt frequency, and process scheduling queue length. The key performance indicators are core parameters quantified from the system operation status data that directly reflect switching efficiency, such as mode switching latency, task response time jitter, and rendering frame rate drop. The abnormal fluctuation features are abrupt patterns in the key performance indicator time-series data that deviate from their normal baseline, such as pulse spikes in response time and periodic stuttering in frame rate. The latency feature vector is a multi-dimensional mathematical representation that can comprehensively characterize a switching latency event, formed by aggregating and vectorizing multiple abnormal fluctuation features according to time windows. The resource dependence is the quantitative correlation between the latency pattern represented in the latency feature vector and the intensity of use of specific computing, storage, or communication resources.

[0055] Furthermore, the system operation status data can be cleaned and aligned using a combination of sliding time windows and Z-score normalization to obtain high-quality system status time-series data. Key performance indicators (KPIs) can be identified from the cleaned data using a combination of rule-based and statistical analysis. For example, the start of a switchover can be marked when the inter-process communication delay exceeds a preset threshold, and the switchover delay duration can be calculated until the target modal UI is fully rendered. Abnormal fluctuation features in the KPIs can be extracted using wavelet transform modulus maxima to accurately capture instantaneous pulses and gradual trends. The delay feature vector can be constructed by concatenating and normalizing the abnormal features of different indicators on a unified time axis. The resource dependency between the delay feature vector and system resource configuration can be analyzed by establishing a modality-resource dependency matrix, where rows represent different delay events, columns represent different system resources, and matrix element values ​​are determined using Pearson correlation coefficients. Alternatively, mutual information calculations can quantify the dependence of a specific latency mode on specific resources. For example, analysis revealed that in the latency event of switching from voice interaction to AR-HUD navigation, the correlation coefficient between the latency feature vector and the GPU rendering unit utilization rate was as high as 0.85, while the correlation coefficient with the CPU was only 0.2. This indicates that the switching latency has a high dependence on graphics computing resources. Combining the resource dependence and the latency feature vector, the switching latency factors corresponding to the in-vehicle intelligent cockpit are determined. First, based on the modality-resource dependence matrix, the resource or group of resources with the highest resource dependence is located and identified as the bottleneck resource. Second, combined with the specific pattern of the latency feature vector, the load status of the bottleneck resource is judged. If the dependence is high and the latency feature shows a continuous high load, the latency factor is determined to be insufficient computing resources. If the dependence is high but the latency feature shows instantaneous peaks and resource contention, the latency factor is determined to be poor process scheduling strategy or resource lock contention, thereby accurately locating the root cause of the switching latency.

[0056] S5. Based on the interaction priority, the interaction intent, and the switching delay factor, generate multi-mode interaction control commands for the in-vehicle intelligent cockpit to execute the user's multi-mode interaction control processing in the in-vehicle intelligent cockpit and obtain control results.

[0057] This invention generates multi-modal interaction control commands for an in-vehicle intelligent cockpit based on the interaction priority, the interaction intent, and the switching delay factor. This executes the user's multi-modal interaction control processing within the in-vehicle intelligent cockpit, obtaining control results. This improves the accuracy of multi-modal interaction control for the user within the in-vehicle intelligent cockpit. Furthermore, the specific steps for generating multi-modal interaction control commands based on the interaction priority, the interaction intent, and the switching delay factor are: first, analyzing the task complexity and real-time requirements in the interaction intent... The process involves several steps: first, defining the core interaction task execution sequence; second, analyzing the task weight distribution in the interaction priority to identify the security boundaries of key interaction paths; third, evaluating the interaction channel carrying capacity under the current system state in conjunction with the switching delay factor; and finally, formulating a modal coupling strategy, resource scheduling scheme, and interaction process optimization mechanism to form a complete cockpit multi-modal interaction control scheme. Based on this scheme, multi-modal interaction control commands for the in-vehicle intelligent cockpit are generated to execute the user's multi-modal interaction control processing within the in-vehicle intelligent cockpit and obtain the control result. For a more intuitive understanding of the multi-modal interaction control command generation process for the in-vehicle intelligent cockpit in this application, please refer to [reference needed]. Figure 2 The diagram shown is a schematic of the instruction generation process in the multi-modal interactive control method for in-vehicle intelligent cockpit provided by the present invention. It should be noted that in the present invention... Figure 2 The flowchart presented is only for the instruction generation process of the in-vehicle intelligent cockpit multi-mode interaction control method, and is not limited to the instruction generation of the in-vehicle intelligent cockpit multi-mode interaction control method in different actual application scenarios.

[0058] Compared to the problems described in the background art, this invention, by performing atomized decomposition processing on the multimodal interaction signals, can accurately separate the core semantic units in voice, touch, and visual signals, avoiding semantic confusion or interference from misoperations, thereby improving the accuracy of subsequent analysis of the interaction intent. This invention, by calculating the intent coupling coefficient between the interaction intents, can quantify the correlation strength and mutual influence between different intents, providing a data foundation for constructing an interaction fusion weight matrix. Furthermore, this invention, by identifying the modal interaction scenarios in the current in-vehicle environment of the intelligent cockpit, can obtain the current environmental interaction context of the intelligent cockpit, and by calculating the recognition confidence level corresponding to the modal interaction scenarios, can verify the reliability of the scene recognition results. To eliminate low-confidence misjudgments and provide reliable scenario basis for subsequent interaction decisions, this invention monitors the switching response delay during the mode switching process of devices in the in-vehicle smart cockpit. When the switching response delay exceeds a preset delay threshold, it analyzes the corresponding switching delay factors in the in-vehicle smart cockpit, accurately identifying factors affecting the smoothness of interaction. This provides an accurate basis for generating subsequent multi-modal interaction control commands. Finally, based on the interaction priority, the interaction intent, and the switching delay factors, this invention generates multi-modal interaction control commands for the in-vehicle smart cockpit to execute the user's multi-modal interaction control processing in the in-vehicle smart cockpit and obtain control results, thereby improving the accuracy of multi-modal interaction control in the in-vehicle smart cockpit. Therefore, the in-vehicle smart cockpit multi-modal interaction control method and system provided by this invention can improve the accuracy of multi-modal interaction control in in-vehicle smart cockpits.

[0059] like Figure 3 The diagram shown is a functional block diagram of a multi-mode interactive control system for an in-vehicle intelligent cockpit according to the present invention.

[0060] The in-vehicle intelligent cockpit multi-modal interactive control system 300 described in this invention can be installed in an electronic device. Depending on the functions implemented, the in-vehicle intelligent cockpit multi-modal interactive control system may include an intent parsing module 301, a fusion matrix construction module 302, an interaction priority determination module 303, a delay factor analysis module 304, and a control command generation module 305. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0061] In this embodiment of the invention, the functions of each module / unit are as follows: The intent parsing module 301 is used to collect multimodal interaction signals generated by the user in the in-vehicle intelligent cockpit, perform atomization decomposition processing on the multimodal interaction signals to obtain atomic-level modal data, and analyze the interaction intent corresponding to the atomic-level modal data. The fusion matrix construction module 302 is used to calculate the intent coupling coefficient between the interactive intents based on the atomic-level modal data, so as to construct the interaction fusion weight matrix between the interactive intents; The interaction priority determination module 303 is used to identify the modal interaction scenarios in the current vehicle environment of the in-vehicle intelligent cockpit, calculate the recognition confidence level corresponding to the modal interaction scenario, and combine the interaction fusion weight matrix to determine the interaction priority of the interaction intention in the modal interaction scenario. The delay factor analysis module 304 is used to monitor the switching response delay of the device mode switching process in the vehicle-mounted intelligent cockpit, and analyze the switching delay factor corresponding to the vehicle-mounted intelligent cockpit when the switching response delay is higher than a preset delay threshold. The control command generation module 305 is used to generate multi-mode interactive control commands for the in-vehicle intelligent cockpit based on the interaction priority, the interaction intent, and the switching delay factor, so as to execute the multi-mode interactive control processing of the user in the in-vehicle intelligent cockpit and obtain the control result.

[0062] In detail, the modules in the in-vehicle intelligent cockpit multi-mode interactive control system 300 described in this embodiment of the invention employ the same methods as described above during use. Figure 1 The method uses the same technical means as the multi-mode interactive control method for in-vehicle intelligent cockpit described in the article, and can produce the same technical effect, so it will not be elaborated here.

[0063] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0064] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-mode interactive control method for an in-vehicle intelligent cockpit, characterized in that, The method includes: The system collects multimodal interaction signals generated by users in the in-vehicle intelligent cockpit, performs atomization decomposition processing on the multimodal interaction signals to obtain atomic-level modal data, and analyzes the interaction intent corresponding to the atomic-level modal data. Based on the atomic-level modal data, the intent coupling coefficient between the interactive intents is calculated to construct the interaction fusion weight matrix between the interactive intents; Identify the modal interaction scenarios in the current in-vehicle environment of the intelligent cockpit, calculate the identification confidence level corresponding to the modal interaction scenario, and combine the interaction fusion weight matrix to determine the interaction priority of the interaction intention in the modal interaction scenario; Monitor the switching response delay of the device mode switching process in the vehicle-mounted intelligent cockpit, and when the switching response delay is higher than a preset delay threshold, analyze the corresponding switching delay factors of the vehicle-mounted intelligent cockpit. Based on the interaction priority, the interaction intent, and the switching delay factor, multi-mode interaction control commands for the in-vehicle intelligent cockpit are generated to execute the multi-mode interaction control processing of the user in the in-vehicle intelligent cockpit and obtain control results.

2. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 1, characterized in that, The step of atomizing the multimodal interaction signal to obtain atomic-level modal data includes: The multimodal interaction signals are decoupled to obtain single-modal stream data; The single-modal stream data is subjected to frame slicing to obtain a time-series frame sequence; The time-series frame sequence is subjected to feature extraction processing to obtain the original feature set; The original feature set is subjected to tensor construction processing to obtain the feature tensor; The feature tensor is subjected to primitive decomposition to obtain modal primitives; The modal primitives are subjected to atomic recombination processing to obtain atomic-level modal data.

3. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 2, characterized in that, The step of performing feature extraction processing on the time-series frame sequence to obtain the original feature set includes: The time-series frame sequence is subjected to multi-scale decomposition to obtain scale feature components; The scale feature components are sparsely encoded to obtain sparse feature vectors; The sparse feature vector is subjected to feature enhancement processing to obtain enhanced sparse features; Topological feature extraction is performed on the enhanced sparse features to obtain topological enhanced features; The topology enhancement features are subjected to feature fusion processing to obtain a fused feature vector; Calculate the feature entropy corresponding to the fused feature vector; Based on the feature entropy, the fused feature vector is filtered to obtain the original feature set.

4. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 1, characterized in that, The analysis of the interaction intent corresponding to the atomic-level modal data includes: Extract the modal data identifiers corresponding to the atomic-level modal data, perform semantic parsing on the modal data identifiers, and obtain the initial modal semantics; The semantic relationships between the initial modal semantics are analyzed, and the initial modal semantics are subjected to intent evolution processing to obtain the intent evolution trajectory; Based on the semantic association, the intent evolution trajectory is clustered to obtain a clustered intent set; Calculate the cluster density corresponding to each intent in the clustering intent set; Based on the cluster density, the core clustering intents in the clustering intent set are extracted; Logical verification is performed on the core clustering intent to obtain the interaction intent corresponding to the atomic-level modality data.

5. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 1, characterized in that, The calculation of the intent coupling coefficient between the interactive intents based on the atomic-level modal data includes: Based on the interaction intent, the atomic-level modal data is aligned to obtain intent modal data; Principal component analysis is performed on the intent modality data to obtain intent modality components; Construct the intent covariance matrix corresponding to the intent modality components; Based on the intent covariance matrix, the modal contribution corresponding to the interaction intent is calculated; Based on the modal contribution, the intent coupling coefficient between the interactive intents is calculated.

6. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 5, characterized in that, The step of calculating the intent coupling coefficient between the interactive intents based on the modal contribution includes: The interaction intent is vectorized to obtain an intent feature vector. Calculate the vector similarity between the intent feature vectors; Based on the vector similarity, determine the intent projection score between the interaction intents; The intent coupling coefficient between the interactive intents is calculated by combining the intent projection score and the modality contribution.

7. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 1, characterized in that, The calculation of the recognition confidence score corresponding to the modal interaction scenario includes: Extract scene environment data from the in-vehicle intelligent cockpit, clean the scene environment data, and obtain target scene environment data; Extract the scene semantic description corresponding to the target scene environment data, and extract the core scene elements from the scene semantic description; The core scene elements are subjected to label mapping processing to obtain scene labels; Based on the scene labels, calculate the recognition confidence level corresponding to the modal interaction scene.

8. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 7, characterized in that, The extraction of core scene elements from the scene semantic description includes: The semantic description of the scene is subjected to structured parsing to obtain semantic feature elements; Calculate the scene core degree corresponding to the semantic feature element, and based on the scene core degree, filter out the key feature elements in the semantic feature element; Analyze the semantic relationships between the key feature elements, and based on the semantic relationships, mine the association and combination patterns between the key feature elements; Based on the associated combination pattern and the key feature elements, the core scene elements in the scene semantic description are determined.

9. The in-vehicle intelligent cockpit multi-mode interactive control method as described in claim 1, characterized in that, The analysis of the switching delay factors corresponding to the in-vehicle intelligent cockpit includes: Collect system operation status data during mode switching, and identify key performance indicators of the in-vehicle intelligent cockpit during mode switching based on the system operation status data; Extract the abnormal fluctuation features from the key performance indicators, and construct the delay feature vector of the in-vehicle intelligent cockpit during mode switching based on the abnormal fluctuation features; Analyze the resource dependency between the aforementioned delay feature vector and the system resource configuration of the in-vehicle intelligent cockpit; By combining the resource dependency and the latency feature vector, the switching latency factor corresponding to the in-vehicle intelligent cockpit is determined.

10. A multi-mode interactive control system for an in-vehicle intelligent cockpit, characterized in that, The system includes: The intent parsing module is used to collect multimodal interaction signals generated by users in the in-vehicle intelligent cockpit, perform atomization decomposition processing on the multimodal interaction signals to obtain atomic-level modal data, and analyze the interaction intent corresponding to the atomic-level modal data. The fusion matrix construction module is used to calculate the intent coupling coefficient between the interactive intents based on the atomic modal data, so as to construct the interaction fusion weight matrix between the interactive intents; The interaction priority determination module is used to identify the modal interaction scenarios in the current vehicle environment of the in-vehicle intelligent cockpit, calculate the recognition confidence corresponding to the modal interaction scenario, and combine the interaction fusion weight matrix to determine the interaction priority of the interaction intention in the modal interaction scenario. The delay factor analysis module is used to monitor the switching response delay of the device mode switching process in the vehicle-mounted intelligent cockpit. When the switching response delay is higher than a preset delay threshold, the module analyzes the corresponding switching delay factors of the vehicle-mounted intelligent cockpit. The control command generation module is used to generate multi-mode interactive control commands for the in-vehicle smart cockpit based on the interaction priority, the interaction intent, and the switching delay factor, so as to execute the multi-mode interactive control processing of the user in the in-vehicle smart cockpit and obtain the control result.

Images