Vehicle-mounted UI adaptive optimization method and system based on multi-modal behavior analysis

Abstract

The invention provides a vehicle-mounted UI adaptive optimization method based on multi-modal behavior analysis, and the method comprises the steps: calculating a plurality of quantitative indexes related to UI interaction experience based on the multi-modal behavior data, collected in real time, of a driver; the multi-modal behavior data comprises facial expressions, voices and intonations and a touch operation sequence; the quantitative indexes comprise an emotion index, an operation efficiency index and a function use range; and if at least one of the emotion index, the operation efficiency index and the function use range of the target vehicle meets the corresponding preset optimization condition, generating a UI fine tuning scheme for the target vehicle based on the met quantitative index, and generating a differential packet to be deployed on the target vehicle in time. The optimization decision-making right is issued to the bicycle end, and lightweight differential package deployment is adopted, so that instant response to individual user experience problems is realized. The traditional OTA upgrading period is shortened from the month level to the day level or even the same day, the problems of optimization lag and individuation deficiency are solved, and the user satisfaction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle UI, in particular to a vehicle UI adaptive optimization method and system based on multi-modal behavior analysis. BACKGROUND

[0002] Currently, the optimization and iteration of the vehicle user interface mainly rely on the following ways, and the inherent defects are as follows: (1) The existing scheme seriously depends on the user's active scoring, questionnaire survey or complaint channel to obtain feedback. This way cannot capture the unconscious negative emotions of the user in the use process, such as confusion and irritability, resulting in a large amount of implicit dissatisfaction being missed. According to statistics, the implicit dissatisfaction missed detection rate of the traditional way can be as high as 62%, so that a large number of real design defects cannot be identified and repaired.

[0003] (2) The improvement of UI usually needs to accumulate a large amount of user data, and then the periodic OTA upgrade package is pushed in batches. From problem discovery, data collection, development testing to final pushing, the whole process usually takes several weeks or even months. It is impossible to respond to high-frequency or urgent interactive problems encountered by individual users in real time, and the user experience problem continues to exist.

[0004] (3) The mainstream optimization scheme tends to provide unified UI design and interaction logic for all users, and fails to adapt to the individual habits of different users. Such a one-size-fits-all design is difficult to meet the needs of the user group. SUMMARY

[0005] The embodiments of the present application provide a vehicle UI adaptive optimization method and system based on multi-modal behavior analysis, to solve the technical problem that the existing technology adopts the OTA upgrade mode of centralized processing to update the UI, resulting in a long optimization cycle and the inability to respond to individual user differentiated UI experience in real time.

[0006] In a first aspect, a vehicle UI adaptive optimization method based on multi-modal behavior analysis is provided, which includes: based on the real-time collected multi-modal behavior data of the driver, calculating a plurality of quantitative indicators related to the UI interaction experience; the multi-modal behavior data includes facial expressions, voice tones and touch operation sequences; the quantitative indicators include emotion indicators, operation efficiency indexes and function use breadth; If at least one of the emotion indicators, operation efficiency indexes and function use breadth of the target vehicle meets the corresponding preset optimization condition, an UI fine-tuning scheme for the target vehicle is generated based on the satisfied quantitative indicators, and a differential package is generated and deployed to the target vehicle in time.

[0007] In some embodiments, after real-time collection of the multi-modal behavior data of the driver, before calculating the quantitative indicators, the multi-modal behavior data is preprocessed, which includes the following steps: The video stream data corresponding to the facial expression, the audio stream data corresponding to the speech tone, and the event stream data corresponding to the touch operation sequence are time-synchronized by using a precise clock protocol, so as to realize alignment of the expression, the speech, and the event. The touch coordinates data corresponding to the touch operation sequence and the user gaze point coordinates extracted from the facial expression data are spatially calibrated by using Zhang's calibration method, so as to establish a mapping relationship between the touch behavior and the visual attention point. The original data of the facial expression, the speech tone, and the touch operation sequence are respectively filtered by using a Kalman filtering algorithm, so as to eliminate environmental noise and signal interference.

[0008] In some embodiments, the emotion index, the operation efficiency index, and the function use breadth are calculated, which includes the following steps: The emotion index is obtained by performing feature extraction and fusion analysis based on the facial expression and the speech tone. The operation efficiency index is obtained by calculating the ratio of the number of steps of the actual operation path of the user to the number of steps of the pre-stored ideal path. The function use breadth is obtained by calculating the ratio of the number of function items that have been activated and used by the user to the total number of function items provided by the system.

[0009] In some embodiments, if at least one of the emotion index, the operation efficiency index, and the function use breadth of the target vehicle satisfies a corresponding preset optimization condition, a UI fine-tuning scheme for the target vehicle is generated based on the satisfied quantitative index, which includes the following steps: If the number of operation failures of a specific UI element exceeds a preset failure threshold value, and at the same time the value of the emotion index exceeds a preset emotion threshold value, it is determined that the optimization condition is satisfied, and a UI fine-tuning scheme for optimizing the layout, size, or interaction logic of the specific UI element is generated; If the operation efficiency index of a specific function path continuously falls below a preset efficiency threshold value within a preset time period, it is determined that the optimization condition is satisfied, and a UI fine-tuning scheme for simplifying the function path or adding a shortcut access entry is generated; If the function use breadth of a target function continuously falls below a preset use rate threshold value, and at the same time a sentence in which the user queries or is confused about the target function is recognized through speech analysis, it is determined that the optimization condition is satisfied, and a UI fine-tuning scheme for improving the visibility of the function or providing a guide prompt is generated.

[0010] In some embodiments, the preset failure threshold value, the emotion threshold value, the efficiency threshold value, and the use rate threshold value can be adjusted according to the real-time driving scene, which includes the following steps: In a congestion scenario, the value of the efficiency threshold value is reduced, and the optimization of the shortcut of the high-frequency operation function is preferentially triggered; In a high-speed scenario, the values of the failure threshold and the emotion threshold are reduced to preferentially trigger fault tolerance optimization of core UI elements related to driving safety. In a night scenario, the value of the usage rate threshold is increased, and the ambient light data is associated to preferentially trigger comfort optimization of interface contrast and brightness.

[0011] In some embodiments, if the deployment proportion of the deployed UI fine-tuning scheme in the vehicle model group reaches a first proportion threshold, a cloud-level global optimization process is started, a standardized UI upgrade scheme for the vehicle model group is generated based on the UI fine-tuning scheme, and batch deployment is performed through an OTA upgrade package.

[0012] In some embodiments, generating a standardized UI upgrade scheme for the vehicle model group further includes differential packaging, which includes the following steps: Based on the hardware performance parameters of the target vehicle group, multiple resource adaptation versions of the same UI upgrade scheme are generated; for the version with high-performance hardware configuration, a UI package containing high-resolution rendering resources and complex animations is deployed, and for the version with standard-performance hardware configuration, a UI package using standard-resolution resources and simplified animations is deployed.

[0013] In some embodiments, after deployment or batch deployment, a feedback loop is further included, which includes the following steps: Monitoring user interaction feedback on the optimized UI; If a user performs a preset version rollback operation, the context data at the time of the rollback operation is recorded; The recorded context data is uploaded to a cloud analysis system for iterative updating of subsequent optimization strategy generation.

[0014] In some embodiments, generating a UI fine-tuning scheme for a target vehicle includes the following steps: Calling a generative artificial intelligence model to generate multiple candidate UI fine-tuning schemes that meet the preset brand design specifications and interaction logic constraints; Using a reinforcement learning model, the predicted operation efficiency improvement and emotion index improvement are used as reward functions to quantitatively evaluate the multiple candidate UI fine-tuning schemes; according to the evaluation scores, the candidate scheme with the highest score is selected as the final UI fine-tuning scheme.

[0015] In a second aspect, the present application proposes a vehicle-mounted UI self-adaptive optimization system based on multi-modal behavior analysis, which includes: A data acquisition and processing module is used to calculate multiple quantitative indicators related to UI interaction experience based on real-time collected multi-modal behavior data of the driver; the multi-modal behavior data includes facial expressions, voice tones, and touch operation sequences; the quantitative indicators include emotion indicators, operation efficiency indexes, and function usage breadth. a deployment module configured to, if at least one of the emotion index, the operation efficiency index, and the function use breadth of the target vehicle meets a corresponding preset optimization condition, generate a UI fine-tuning scheme for the target vehicle based on the satisfied quantitative index, and generate a differential package and deploy the differential package to the target vehicle in time.

[0016] The technical scheme provided in the application has the following beneficial effects: The application provides a vehicle-mounted UI self-adaptive optimization method based on multi-modal behavior analysis, wherein the decision-making right of optimization triggering is decentralized to the single vehicle level. The system directly calculates and judges at the vehicle end based on the real-time collected data. As long as the experience data of a single user reaches the preset standard, the optimization process can be triggered immediately, without waiting for the cloud to discover the group trend, which fundamentally changes the face-to-point lag logic and realizes the point-starting immediate response. The deployment content is a differential package. The differential package only contains the changed part, such as the icon and position parameter of a certain button, and the volume is much smaller than the complete OTA package, so the purpose of timely deployment can be achieved. This lightweight deployment method makes the cost of optimizing a single user extremely low, thereby having technical and economic feasibility. Since there is no need to wait for data reporting and cloud analysis, the optimization process can be started in a short time after the problem occurs, combined with the rapid deployment of the differential package, the repair cycle of high-frequency problems can be greatly shortened from the traditional one month or more to several days or even the same day; the system can customize the repair for unique problems encountered by different users or the same user in different scenarios. The technical problem that the optimization cycle is long and the individual user's differentiated UI experience cannot be responded to immediately due to the use of the centralized processing OTA upgrade mode to update the UI in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The application provides a vehicle-mounted UI self-adaptive optimization method based on multi-modal behavior analysis. Figure 2 The application provides a vehicle-mounted UI self-adaptive optimization method based on multi-modal behavior analysis. DETAILED DESCRIPTION

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0020] To make the technical problem that this application aims to solve clearer, the causes of the technical problem will be analyzed in detail below.

[0021] Current in-vehicle UI optimization technologies suffer from problems such as passive feedback, long optimization cycles, and lack of personalization, which are analyzed in detail below: The current system suffers from passive feedback and a high rate of missed implicit complaints. Existing technological channels, such as ratings and customer service complaints, require users to interrupt their current task and actively express their dissatisfaction. However, users in driving scenarios experience high cognitive loads, and most minor frustrations are ignored or forgotten, resulting in a large number of negative experiences going unrecorded. Furthermore, the current system fails to recognize the need for real-time correlation analysis of user facial expressions like frowns and sighs, as well as voice signals, with repeated, ineffective clicks. This prevents the system from interpreting a series of unconscious physiological and behavioral reactions as expressions of dissatisfaction with specific UI designs.

[0022] The optimization cycle is long and lacks immediate response. From the moment a single user encounters a problem, to the manufacturer collecting enough data to confirm it's a common issue, and then to development, testing, packaging, and push notifications, the entire process is lengthy and cannot meet the urgent needs of individual users. Traditional architectures treat vehicles as purely data collection and solution execution terminals, with all complex analysis and decision-making occurring in the cloud. This architecture inevitably leads to response delays. Even if an individual problem is identified in the cloud, preparing a separate OTA update for it is too costly and impractical. The root cause lies in the lack of an intelligent agent deployed on the vehicle that can understand the user and make lightweight optimization decisions.

[0023] The strategy is generic, lacking personalization. To ensure stability and control costs, manufacturers tend to provide a uniform UI design for all users. So-called personalization is often limited to superficial levels like theme colors and wallpapers, failing to delve into core experience aspects such as interaction logic, layout, and information density. Traditional UI design is static, fixed once manufactured. It cannot sense whether the current situation is highway driving or traffic jam waiting, nor can it learn whether user A prefers voice operation or user B prefers touch. The core issue is the lack of a continuous evaluation-optimization-validation closed loop. Even with limited data, there is a lack of effective algorithmic models to dynamically generate and validate optimal personalized solutions based on individual data, while ensuring compliance with safety and brand guidelines.

[0024] In summary, the root cause of the problems in existing technologies lies in their centralized architecture, passive perception, lagging decision-making, and static optimization. This application addresses these issues by proposing a new paradigm that is distributed, proactively perceiving, making real-time decisions, and dynamically evolving, thereby fundamentally changing traditional solutions and systematically resolving the aforementioned pain points.

[0025] In a first aspect, embodiments of this application provide an adaptive optimization method for in-vehicle UI, referring to... Figure 1 , Figure 1 This is a schematic flowchart of an in-vehicle UI adaptive optimization method based on multimodal behavior analysis, provided as an embodiment of this application. Figure 1 As shown, an adaptive optimization method for in-vehicle UI based on multimodal behavior analysis includes: S100: Based on real-time collected multimodal behavioral data of drivers, calculates multiple quantitative indicators related to UI interaction experience; multimodal behavioral data includes facial expressions, voice tone and touch operation sequences; quantitative indicators include emotion indicators, operation efficiency index and function usage breadth; S200. If at least one of the target vehicle's emotional index, operational efficiency index, and functional usage breadth meets the corresponding preset optimization conditions, then a UI fine-tuning scheme for the target vehicle is generated based on the met quantitative indicators, and a differential package is generated and deployed to the target vehicle in a timely manner.

[0026] This approach decentralizes the decision-making power for optimization triggering to the individual vehicle level. The system performs calculations and judgments directly on the vehicle based on real-time collected data. As long as a single user's experience data meets the preset standards, the optimization process can be triggered immediately, without waiting for the cloud to detect a group trend. This fundamentally changes the lagging logic from surface to point, achieving instant response initiated from a single point. The deployment content is a differential package. The differential package only contains the changed parts, such as the icon or position parameters of a button, and its size is much smaller than the complete OTA package, thus achieving the goal of timely deployment. This lightweight deployment method makes the cost of optimizing for a single user extremely low, thus making it feasible from both a technical and economic perspective. Since there is no need to wait for data reporting and cloud analysis, the optimization process can be initiated shortly after a problem occurs. Combined with the rapid deployment of differential packages, the repair cycle for high-frequency problems can be significantly shortened from the traditional month or longer to several days or even the same day. The system can perform customized repairs for unique problems encountered by different users or even the same user in different scenarios. This solves the technical problem that existing technologies use a centralized OTA upgrade mode to update the UI, resulting in long optimization cycles and an inability to respond instantly to the differentiated UI experiences of individual users.

[0027] In some preferred embodiments, after real-time acquisition of the driver's multimodal behavior data and before calculating quantitative indicators, the method further includes preprocessing the multimodal behavior data, which includes the following steps: A precise clock protocol is used to perform time synchronization processing on the video stream data corresponding to facial expressions, the audio stream data corresponding to voice intonation, and the event stream data corresponding to touch operation sequences, so as to achieve alignment of expressions, voice and events; Using Zhang's calibration method, the touch coordinate data corresponding to the touch operation sequence is spatially calibrated with the user gaze point coordinates extracted from the facial expression data to establish a mapping relationship between touch behavior and visual attention point. The raw data of facial expressions, voice intonation, and touch operation sequences are filtered using the Kalman filter algorithm to eliminate environmental noise and signal interference.

[0028] In this embodiment, high-precision time synchronization is achieved through a precision clock protocol, ensuring that the user's facial expressions, voice, and operations are aligned on a strict timeline. This provides the possibility for accurate causal analysis, such as how misoperations immediately trigger negative emotions, thus improving the credibility of quantitative indicators and decisions. The Zhang calibration method is used to establish a mapping between the visual and touch coordinate systems, enabling the system to accurately determine whether the user's gaze focus aligns with the operation target. This is a crucial prerequisite for diagnosing UI defects, such as buttons that are too small leading to accidental touches or unreasonable layouts causing gaze jumps. Kalman filtering and other algorithms are used for noise reduction, effectively filtering out environmental interference and extracting pure behavioral signals that reflect the user's true intentions. This significantly reduces false alarms and missed alarms caused by data quality issues, allowing the system to remain stable in complex real-world driving environments.

[0029] In some preferred embodiments, calculating sentiment indicators, operational efficiency indices, and feature usage breadth includes the following steps: Based on facial expressions and voice intonation, feature extraction and fusion analysis are performed to obtain emotion indicators; The operation efficiency index is obtained by calculating the ratio of the number of steps in the user's actual operation path to the number of steps in the pre-stored ideal path. The breadth of user usage is determined by comparing the number of activated and used features with the total number of features provided by the system.

[0030] In this embodiment, three key indicators—emotion index (E), operational efficiency index (Ef), and functional usage breadth (U)—are defined to provide a comprehensive, data-driven quantification of the UI experience from the perspectives of emotion (feeling), behavior (rationality), and cognition (exploration). This approach eliminates reliance on subjective feedback and establishes a complete experience evaluation system. Low values ​​for different indicators directly point to different types of UI defects. A low emotion index may indicate visual or interaction design issues; a low operational efficiency index directly suggests cumbersome task processes; and a low functional usage breadth indicates hidden function entry points or high learning costs. This multi-dimensional measurement ensures that subsequent optimization is not blind but rather has clear and targeted objectives. Specifically, micro-expression recognition uses an improved VGG16 network, and voice emotion analysis uses the SKEP Chinese emotion analysis model. In some preferred embodiments, if at least one of the target vehicle's emotion index, operational efficiency index, and functional usage breadth meets the corresponding preset optimization conditions, a UI fine-tuning scheme for the target vehicle is generated based on the met quantitative indicators, which includes the following steps: If the number of failed operations for a specific UI element exceeds a preset failure threshold, and at the same time the value of the sentiment index exceeds a preset sentiment threshold, then the optimization conditions are met, and a UI fine-tuning plan is generated to optimize the layout, size, or interaction logic of that specific UI element. If the operational efficiency index of a specific function path remains below the preset efficiency threshold for a preset time period, it is determined that the optimization conditions are met, and a UI fine-tuning scheme is generated to simplify the function path or add a quick access entry. If the usage breadth of the target function is consistently lower than the preset usage rate threshold, and at the same time, the voice analysis identifies user queries or statements that confuse the target function, then the optimization conditions are met, and a UI tweaking scheme is generated to improve the visibility of the function or provide guidance prompts.

[0031] In this embodiment, a set of strongly correlated triggering rules is established. For example, Rule 1 requires both operation failure and negative emotion to be met simultaneously, which greatly increases the confidence in diagnosing design-misleading problems. This combination of operational frustration and emotional triggering strongly indicates that the root cause of the problem lies in the UI itself, rather than accidental user error. This significantly reduces the risk of incorrect optimization; each triggering condition directly corresponds to a specific UI defect and optimization direction. Rule 1 targets specific UI elements, Rule 2 targets functional paths, and Rule 3 targets functional discoverability. This refined classification and diagnostic mechanism ensures that the generated optimization solutions are highly targeted; it covers the most important types of problems in UI design: usability (Rule 1), ease of use (Rule 2), and discoverability (Rule 3), enabling the system's adaptive optimization capabilities to cover the vast majority of user experience pain points, becoming a comprehensive and intelligent solution.

[0032] In some preferred embodiments, the preset failure threshold, emotion threshold, efficiency threshold, and usage rate threshold can be adjusted according to the real-time driving scenario, which includes the following steps: In congested scenarios, lowering the efficiency threshold prioritizes optimizing high-frequency functions for faster access; in high-speed scenarios, lowering the failure and emotional thresholds prioritizes optimizing core UI elements related to driving safety for greater fault tolerance; in nighttime scenarios, increasing the usage threshold and linking it to ambient light data prioritizes optimizing interface contrast and brightness for greater comfort.

[0033] In this embodiment, the system no longer adopts a one-size-fits-all approach but possesses scene awareness and dynamic strategy adjustment capabilities. By dynamically adjusting thresholds, the system changes its focus and sensitivity in different scenarios, ensuring that the optimization strategy is highly consistent with the user's core needs in the current scenario—a higher level of user experience optimization. In high-speed scenarios, by lowering the trigger thresholds for operational failures and negative emotions, the system can more sensitively detect and fix UI issues that may lead to driver distraction, frustration, or misoperation, reflecting the principle of safety first. In congested scenarios, drivers may interact with the UI more frequently, so high-frequency functions are prioritized for optimization. In nighttime scenarios, the usage rate of non-core functions is ignored, focusing instead on visual comfort. This scenario-priority-based resource allocation ensures that each optimization generates maximum user value in the present moment, improving the overall intelligence and effectiveness of the system.

[0034] In some preferred embodiments, if the deployment ratio of the UI tweaking scheme in the corresponding vehicle model group reaches a first percentage threshold, a cloud-based global optimization process is initiated to generate a standardized UI upgrade scheme for that vehicle model group based on the UI tweaking scheme, and then deploy it in batches via OTA upgrade packages.

[0035] In this embodiment, by introducing a deployment ratio threshold as a group verification mechanism, it is ensured that only solutions that have been fully tested in practice and have universal value can enter the global upgrade process. Traditional OTA upgrades are often based on limited testing and prediction in the laboratory, which cannot truly reflect the usage of a large number of users in complex real-world scenarios, resulting in low user satisfaction after the upgrade or even the need for rollback, which is costly. That is, a solution is globalized because it has proven its value in a sufficiently large user sample (the deployment ratio meets the standard), which greatly reduces the risk of failure and resource waste in global upgrades. It cleverly solves the transition problem from single-vehicle-level optimization to group-level upgrades, not only ensuring the quality and security of global optimization, but also forming an efficient, reliable, and adaptive complete technical solution with single-vehicle-level optimization.

[0036] In some preferred embodiments, generating a standardized UI upgrade solution for this vehicle model group also includes differentiated packaging, which includes the following steps: Based on the hardware performance parameters of the target vehicle group, multiple resource-adapted versions are generated for the same UI upgrade solution; for the version with high-performance hardware configuration, a UI package containing high-resolution rendering resources and complex animation effects is deployed, while for the version with standard performance hardware configuration, a UI package using standard resolution resources and simplified animation effects is deployed.

[0037] In this embodiment, while ensuring the consistency of core functions and interaction logic, multiple resource packages adapted to different hardware performance are automatically generated through a single development process. For high-performance hardware, high-resolution resources and complex animations are provided to fully leverage its hardware potential, delivering top-notch smoothness and visual impact to users. For standard hardware, streamlined resources are provided to ensure smooth operation and avoid lag caused by upgrades. This allows users with different configurations to obtain the best experience matching their hardware performance, improving the satisfaction of all users. A simplified package is provided for low-end models, reducing the computational and storage load during installation, minimizing the risk of upgrade failure or system instability after upgrade due to excessive resources, and ensuring the reliability of the upgrade.

[0038] In some preferred embodiments, after deployment or batch deployment, a feedback loop is also included, which comprises the following steps: Monitor user feedback on the optimized UI; If a user is detected to have performed a preset version rollback operation, the context data at the time of the rollback operation is recorded; the recorded context data is uploaded to the cloud analysis system for iterative updates to the generation of subsequent optimization strategies.

[0039] In this embodiment, the system can not only perceive, analyze, decide, and execute, but also verify and learn, thus becoming a self-evolving system capable of continuous iteration and improvement, rather than a static solution. A user's rollback operation is an extremely strong negative feedback signal. Recording the context data during rollback, such as what operations were performed before the rollback, provides the most direct evidence for analyzing why the optimization solution failed. This high-quality negative sample data is crucial for training and optimizing AI models, effectively preventing the system from repeatedly generating unpopular designs. Uploading rollback data to the cloud for analysis allows manufacturers to discover potential flaws in optimization solutions or unexpected user behavior patterns from a macro perspective. This is not only a self-improvement of the UI optimization system, but also provides valuable, data-driven decision-making basis for the next generation design of the entire product, deeply integrating user feedback into every stage of the product lifecycle. It's important to note that rollback data is uploaded to the cloud in real time via 4G / 5G networks.

[0040] In some preferred embodiments, generating a UI tweak for the target vehicle includes the following steps: A generative artificial intelligence model is invoked to generate multiple candidate UI fine-tuning schemes that conform to preset brand design specifications and interaction logic constraints. A reinforcement learning model is used to quantitatively evaluate the multiple candidate UI fine-tuning schemes, with the predicted improvement in operational efficiency and the improvement in emotional indicators as reward functions. Based on the evaluation scores, the candidate scheme with the highest score is selected as the final UI fine-tuning scheme.

[0041] In this embodiment, generative artificial intelligence is introduced to automate solution generation, freeing designers from repetitive tasks and greatly improving efficiency. Solution generation is closely integrated with effect prediction. During the solution generation stage, a reinforcement learning model is used to conduct forward-looking evaluations based on quantified user experience indicators (operational efficiency, emotion), achieving truly data-driven decision-making and ensuring the effectiveness of optimized solutions. The reinforcement learning model uses the "predicted improvement in operational efficiency and emotion indicators" as reward functions for quantitative evaluation, making the solution selection process data-driven and systematic. The system can select the solution that theoretically best improves user experience, thereby significantly increasing the success rate of optimization measures and user satisfaction, and avoiding the waste of resources caused by blind trial and error.

[0042] It's important to know that data acquisition is completed through a data preprocessing module, and the components involved in the acquisition include: In-vehicle camera: 8-megapixel RGB camera, 30fps frame rate, 120° horizontal field of view, supports real-time extraction of micro-expression feature points, such as drooping corners of the mouth, squinting movements, and gaze point recognition; Microphone array: 4-channel MEMS microphone, sampling rate 48kHz, supports 360° sound source localization and speech emotion feature extraction (volume changes, intonation pauses). Touch sensor: 100Hz sampling rate, supports pressure sensing (0-5N) and multi-finger operation recognition; Vehicle status sensors: CAN bus inputs driving status data such as vehicle speed, steering angle, and gear position.

[0043] For data layer fusion, multi-source data is stitched together based on timestamp alignment to form a 10Hz fused data stream. For feature layer fusion, micro-expression feature vectors (512-dimensional), voice emotion feature vectors (256-dimensional), and operational behavior feature vectors (128-dimensional) are fused through an attention mechanism. For decision layer fusion, multimodal decision fusion is achieved based on DS evidence argumentation. Based on these, the feature extraction module completes multimodal fusion and establishes a connection with the multimodal fusion module. The quantitative index optimization judgment module determines whether the optimization conditions are met, then generates the optimization strategy, further generates a differential update package, and sends it to the in-vehicle terminal. Different UI rendering engines are used based on different configurations. Finally, the user feedback collection module establishes a connection with the multimodal fusion module through rollback tags.

[0044] Secondly, this application provides an in-vehicle UI adaptive optimization system based on multimodal behavior analysis, which includes: The data acquisition and processing module is used to calculate multiple quantitative indicators related to the UI interaction experience based on real-time collected multimodal behavioral data of the driver; the multimodal behavioral data includes facial expressions, voice tone, and touch operation sequences; the quantitative indicators include emotion indicators, operation efficiency index, and function usage breadth; The deployment module is used to generate a UI fine-tuning scheme for the target vehicle based on the quantitative indicators if the quantitative indicators of the target vehicle meet the corresponding preset optimization conditions, and to generate a differential package for timely deployment to the target vehicle.

[0045] By setting up this system, the modular design clearly defines the responsibilities of each functional unit, improving the system's maintainability and scalability; the hardware implementation description provides direct technical guidance for productization, ensuring the practical value of patent protection; the correspondence between functional modules and method steps establishes a complete method-system protection system; clear interface definitions ensure efficient collaboration between modules, improving the overall system's response speed and reliability. It covers all possible hardware and software architectures for implementing this in-vehicle UI adaptive optimization method.

[0046] Thirdly, embodiments of this application provide a vehicle including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements any step of the in-vehicle UI adaptive optimization method.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0048] The present application has a storage medium storing an in-vehicle UI adaptive optimization program, wherein when the in-vehicle UI adaptive optimization program is executed by a processor, it implements the steps of the in-vehicle UI adaptive optimization method as described above.

[0049] The method implemented when the in-vehicle UI adaptive optimization program is executed can be referred to in various embodiments of the in-vehicle UI adaptive optimization method of this application, and will not be repeated here.

[0050] The following are some specific examples in various scenarios: (1) When the driver tried to click the “Home” navigation shortcut button on the central control screen multiple times in congested traffic conditions, the driver failed (by accidentally touching an adjacent button). At the same time, the system detected that the driver had negative expressions such as frowning and sighing. Features such as raised eyebrows and downturned corners of the mouth were extracted from the video stream and combined with the sighing features in the speech to calculate the emotion index of this interaction session, E=0.82 (greater than the threshold of 0.7). The analysis showed that in the four operations of the user on the target button (Go Home), three of them fell outside the effective area, and the operation efficiency index Ef=0.25 (less than the threshold of 0.4). The system detected that the number of operation failures (3 times) was greater than the failure threshold (2 times), and the sentiment index (0.82) was greater than the sentiment threshold (0.7). Since the first trigger condition was met, it was determined to be a UI defect of design misleading nature.

[0051] Based on the brand design guidelines, the generative AI model generates three candidate solutions that enlarge the button size by 40% and adjust the layout; the reinforcement learning model selects the optimal solution with the goal of improving the success rate of operation and reducing negative emotions; through the incremental update protocol, the differential package containing only the update of the button resource is instantly pushed to the driver's vehicle, and the UI is updated quickly without restarting; Due to the congested environment, the system lowered the threshold for operational efficiency, making it more sensitive to such inconveniences and thus quickly triggering this optimization. After optimization, the system detected that the driver successfully clicked the button once, and no further negative expressions appeared during subsequent passage. This tweak was marked as effective.

[0052] (2) The data analysis platform found that in a certain vehicle model group, more than 15% of the vehicles reported that the operation efficiency index of the air conditioning temperature adjustment function was consistently low. Users usually need to enter the secondary menu to complete the adjustment.

[0053] Anonymized data from numerous vehicles was collected in the cloud, revealing that the average operational efficiency index for air conditioning temperature adjustment was 0.35, significantly lower than the efficiency threshold (0.5). Data correlation analysis showed that this low efficiency was directly related to the number of touch steps (average 3 steps). Because this inefficiency is widespread across the vehicle group (exceeding the set first percentage threshold, i.e., 10%), and because similar fine-tuning solutions at the single-vehicle level have been widely adopted and proven effective, the system triggers cloud-level global optimization.

[0054] The UI designer reviewed the design to ensure that adding the temperature quick control bar to the main screen complies with ASIL-B driving safety standards and does not obscure key driving information; the brand's UI gene library was used to ensure that the visual style of the quick control bar is consistent with the family design language of the vehicle series; in a simulation environment, virtual users were simulated to use the new UI in different scenarios such as rain, snow, and night to verify that its operation path heatmap coverage is high and there is no potential risk of accidental touch.

[0055] For vehicles equipped with high-end chips, the update includes smooth temperature-sensitive animations and high-resolution icons; for mainstream configurations, static icons and instant responsiveness are used to ensure smooth performance. This standardized UI upgrade package is pushed out in batches to all vehicles in this model group via the OTA management module.

[0056] (3) After Ms. Zhang's vehicle was updated, a shortcut to the seat massage function was added to the main screen. However, the system subsequently detected that she performed a three-finger swipe down to roll back, and restored the UI to the old version.

[0057] The system detected a preset version rollback operation and immediately recorded the context data within 30 seconds before the rollback, including: operation sequence (the user had clicked the newly added massage shortcut icon once but immediately canceled it); visual attention data (eye tracking revealed that the user's gaze lingered on the icon for a very short time); and vehicle status (the vehicle was in high-speed cruising mode at the time). Contextual data is uploaded to the cloud analysis system, and causal inference analysis is used to infer the root cause. In high-speed driving scenarios, users are resistant to shortcuts to non-core driving functions (such as seat massage) because they believe that they may interfere with their attention, so they choose to disable them.

[0058] This negative feedback data is labeled and used to update the reward function of the reinforcement learning model: in high-speed scenarios, negative weights are applied to UI changes to non-security-related functions.

[0059] When generating optimization plans for other users in the future, the model will take into account the relevance of the scenario and the function, and avoid actively pushing shortcuts to such entertainment and comfort functions in high-speed scenarios, thereby improving the acceptance and effectiveness of the optimization plan.

[0060] The three embodiments above demonstrate the specific applications of this invention at three levels: vehicle-level real-time optimization, cloud-level global optimization, and feedback-based continuous self-learning. They fully embody the rationality of the closed loop from data perception and intelligent decision-making to deployment and verification.

[0061] The beneficial effects of this invention include: This paper proposes an adaptive optimization method, system, vehicle, and readable storage medium for in-vehicle UI. Based on real-time collected multimodal driver behavior data, it calculates multiple quantitative indicators related to UI interaction experience. Instead of passively waiting for user complaints, it proactively and continuously collects the most authentic, unconscious user interaction behaviors, such as frowning, sighing, and repeated accidental touches. By quantifying this objective data, subjective dissatisfaction is transformed into measurable indicators, thus achieving accurate capture of implicit dissatisfaction and solving the problem of passive feedback. For sudden and urgent UI problems in individual vehicles, such as a button being difficult to click, a fine-tuning solution is immediately generated and deployed via an incremental update protocol once the indicators meet the criteria. This deployment method is lightweight and fast, achieving rapid response updates and greatly shortening the optimization cycle, solving the problem of response lag. For common problems, traditional OTA batch deployment is only initiated after the deployment ratio of fine-tuning solutions reaches the target. This avoids unnecessary global updates and ensures the stability of mainstream optimizations. This application distinguishes between UI fine-tuning solutions for the target vehicle and standardized UI upgrade solutions for the same vehicle model group. The starting point for optimization decisions is the target vehicle, that is, first considering the unique problems of individual users to generate personalized fine-tuning solutions. To ensure that optimizations can adapt to individual differences and specific scenarios, personalized solutions will not be blindly promoted. Only after their effectiveness has been verified by a group, i.e., the deployment ratio reaches a threshold, will they be solidified into standardized solutions for promotion. This point-to-surface approach ensures that the global strategy originates from a large number of individual practices and is a verified, superior, and universal strategy. It solves the technical problems of response lag and lack of personalization caused by existing in-vehicle UI optimization methods that rely on passive feedback, batch updates, and universal strategies.

[0062] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0063] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0064] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0065] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0066] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0068] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for adaptive optimization of in-vehicle UI based on multimodal behavior analysis, characterized in that, It includes: Based on real-time collected multimodal behavior data of drivers, multiple quantitative indicators related to UI interaction experience are calculated; The multimodal behavioral data includes facial expressions, voice tone, and touch operation sequences; the quantitative indicators include emotion indicators, operational efficiency index, and functional usage breadth. If at least one of the target vehicle's emotion index, operational efficiency index, and functional usage breadth meets the corresponding preset optimization conditions, a UI fine-tuning scheme for the target vehicle is generated based on the met quantitative indicators, and a differential package is generated and deployed to the target vehicle in a timely manner.

2. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 1, characterized in that, After real-time acquisition of the driver's multimodal behavior data and before calculating the quantitative indicators, the method further includes preprocessing the multimodal behavior data, which includes the following steps: A precision clock protocol is used to perform time synchronization processing on the video stream data corresponding to the facial expressions, the audio stream data corresponding to the voice tone, and the event stream data corresponding to the touch operation sequence, so as to achieve alignment of expressions, voice and events; Using Zhang's calibration method, the touch coordinate data corresponding to the touch operation sequence is spatially calibrated with the user gaze point coordinates extracted from the facial expression data to establish a mapping relationship between touch behavior and visual attention point. The raw data of facial expressions, voice intonation, and touch operation sequences are filtered using the Kalman filter algorithm to eliminate environmental noise and signal interference.

3. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 1, characterized in that, Calculating sentiment indicators, operational efficiency index, and feature usage breadth involves the following steps: The emotion index is obtained by feature extraction and fusion analysis based on facial expressions and voice intonation. The operation efficiency index is obtained by calculating the ratio of the number of steps in the user's actual operation path to the number of steps in the pre-stored ideal path. The breadth of user usage is obtained by comparing the number of activated and used features with the total number of features provided by the system and calculating their ratio.

4. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 3, characterized in that, If at least one of the target vehicle's emotion index, operational efficiency index, and functional usage breadth meets the corresponding preset optimization conditions, then a UI fine-tuning scheme for the target vehicle is generated based on the met quantitative indicators, which includes the following steps: If the number of failed operations for a specific UI element exceeds a preset failure threshold, and at the same time the value of the emotion indicator exceeds a preset emotion threshold, then the optimization conditions are met, and a UI fine-tuning scheme is generated to optimize the layout, size, or interaction logic of the specific UI element. If the operational efficiency index of a specific function path remains below the preset efficiency threshold for a preset time period, it is determined that the optimization conditions are met, and a UI fine-tuning scheme is generated to simplify the function path or add a quick access entry. If the usage breadth of the target function is consistently lower than a preset usage rate threshold, and at the same time, voice analysis identifies user queries or statements that confuse the target function, then the optimization conditions are met, and a UI tweaking scheme is generated to improve the visibility of the function or provide guidance prompts.

5. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 4, characterized in that, The preset failure threshold, emotion threshold, efficiency threshold, and usage rate threshold can be adjusted according to the real-time driving scenario, and include the following steps: In congested scenarios, the value of the efficiency threshold is reduced to prioritize the optimization of high-frequency operation functions. In high-speed scenarios, the values ​​of the failure threshold and the emotion threshold are reduced, and fault tolerance optimization of core UI elements related to driving safety is triggered first. In nighttime scenarios, the usage threshold value is increased, and ambient light data is correlated to prioritize comfort optimization of interface contrast and brightness.

6. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 1, characterized in that: If the deployment ratio of the UI tweaking scheme in the corresponding vehicle model group reaches the first percentage threshold, then the cloud-level global optimization process is initiated. Based on the UI tweaking scheme, a standardized UI upgrade scheme for the vehicle model group is generated and deployed in batches via OTA upgrade packages.

7. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 6, characterized in that, Generate a standardized UI upgrade solution for this vehicle model group, which also includes differentiated packaging, and includes the following steps: Based on the hardware performance parameters of the target vehicle group, multiple resource-adapted versions are generated for the same UI upgrade solution; for the version with high-performance hardware configuration, a UI package containing high-resolution rendering resources and complex animation effects is deployed, while for the version with standard performance hardware configuration, a UI package using standard resolution resources and simplified animation effects is deployed.

8. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 1, characterized in that, After deployment or batch deployment, a feedback loop is also included, which includes the following steps: Monitor user feedback on the optimized UI; If a user is detected to be performing a preset version rollback operation, the context data at the time of the rollback operation is recorded. The recorded context data is uploaded to a cloud-based analysis system for iterative updates to the generation of subsequent optimization strategies.

9. The in-vehicle UI adaptive optimization method based on multimodal behavior analysis as described in claim 1, characterized in that, Generating a UI tweaking scheme for the target vehicle includes the following steps: The generative artificial intelligence model is invoked to generate multiple candidate UI fine-tuning schemes that conform to the preset brand design specifications and interaction logic constraints; Using a reinforcement learning model, the predicted improvement in operational efficiency and the improvement in sentiment indicators are used as reward functions to quantitatively evaluate multiple candidate UI fine-tuning schemes; based on the evaluation scores, the candidate scheme with the highest score is selected as the final UI fine-tuning scheme.

10. An adaptive optimization system for in-vehicle UI based on multimodal behavior analysis, characterized in that, It includes: The data acquisition and processing module is used to calculate multiple quantitative indicators related to the UI interaction experience based on real-time collected multimodal behavioral data of drivers. The multimodal behavioral data includes facial expressions, voice tone, and touch operation sequences; the quantitative indicators include emotion indicators, operational efficiency index, and functional usage breadth. The deployment module is used to generate a UI fine-tuning scheme for the target vehicle based on the satisfied quantitative indicators if at least one of the target vehicle's emotion index, operational efficiency index, and functional usage breadth meets the corresponding preset optimization conditions, and to generate a differential package for timely deployment to the target vehicle.

Images