Cockpit multi-modal interaction control method, device, equipment, medium and product

Abstract

This invention belongs to the field of intelligent cockpit technology and discloses a cockpit multimodal interaction control method, device, equipment, medium, and product. The method includes: acquiring user interaction capability characteristic data and historical interaction data, wherein the historical interaction data includes the interaction modal used in each interaction, associated scene data, and interaction performance; dividing the scene data into sub-scenes; for different sub-scenes, determining the priority of the corresponding interaction modal based on the usage frequency and interaction performance of the interaction modal in that sub-scene, obtaining an initial interaction priority matrix; judging the user's capability adaptability to each interaction modal based on the user interaction capability characteristic data, and correcting the initial interaction priority matrix based on the capability adaptability; acquiring current scene data, and determining the highest priority interaction modal in the current sub-scene as the current standby primary modal based on the user interaction priority matrix. This invention can meet the diverse interaction needs of different users.

Description

Technical Field

[0001] This invention belongs to the field of intelligent cockpit technology, and particularly relates to a cockpit multimodal interactive control method, device, equipment, medium and product. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of smart cockpit technology, existing cockpit interaction systems have gradually evolved from single-modal interaction to multi-modal interaction. To ensure driving safety, some systems adjust the activation priority of interaction modes based on the vehicle's driving status and support users to manually switch between different interaction modes to adapt to different operating habits. However, the above-mentioned interaction mode settings do not consider the interaction needs of special groups, such as users with physical or sensory impairments. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention provides a cockpit multimodal interaction control method, device, equipment, medium and product to improve the continuity of interaction in the cockpit.

[0005] To achieve the above objectives, a first aspect of the present invention provides a cockpit multimodal interactive control method, comprising the following steps: Acquire user interaction capability feature data and historical interaction data, wherein the historical interaction data includes the interaction modality used in each interaction, related scene data, and interaction performance; The scene data is divided into sub-scenes. For different sub-scenes, the priority of the interaction modality corresponding to the sub-scene is determined based on the frequency of use and interaction performance of the interaction modality in the sub-scene, and an initial interaction priority matrix is ​​obtained. The user's ability to adapt to each interaction modality is determined based on the user's interaction ability characteristic data, and the initial interaction priority matrix is ​​corrected based on the ability adaptability. Obtain current scene data, determine the current sub-scene, and based on the user interaction priority matrix, determine the highest priority interaction mode in the current sub-scene as the current standby main mode, waiting for user input.

[0006] In some embodiments, the user interaction capability feature data includes the user's physiological capabilities and capability level for adapting to cockpit interaction operations.

[0007] In some embodiments, determining the priority of the interaction modal corresponding to the sub-scenario based on the usage frequency and interaction performance of the interaction modal in the sub-scenario includes: for the historical interaction modal in each sub-scenario, counting the total number of historical uses of each interaction modal in the sub-scenario and calculating the usage frequency ratio of each modal; at the same time, evaluating the interaction performance of each interaction modal in the sub-scenario; and determining the sorting priority based on the usage frequency and interaction performance evaluation results of each modal.

[0008] In some embodiments, modifying the initial interaction priority matrix based on capability adaptability includes: determining the mapping relationship between interaction modalities and required physiological capability levels, assessing the user's capability adaptability to each interaction modality, and verifying the order of each interaction modality in the initial interaction priority matrix based on capability adaptability.

[0009] In some embodiments, in addition to the primary mode, other backup interaction modes are equipped with low-power trigger thresholds. When an input action is detected that meets the trigger threshold of a backup interaction mode, the backup interaction mode is woken up, and the operation intention in the user input signal is parsed. If the operation intention is successfully interpreted and can be matched with the cockpit interaction function, the input signal is considered valid, converted into a cockpit control command and executed, and the standby monitoring of the primary interaction mode is paused.

[0010] In some embodiments, the method further includes: while waiting for user input, monitoring scene data in real time to determine whether the sub-scene has changed; if so, triggering an interaction mode switch; and determining the interaction mode with the highest priority in the current sub-scene as the current standby main mode based on the user interaction priority matrix.

[0011] A second aspect of the present invention provides a cockpit multimodal interactive control device, comprising: The historical data acquisition module is configured to acquire user interaction capability feature data and historical interaction data, wherein the historical interaction data includes the interaction modality used in each interaction, related scene data, and interaction performance. The priority matrix calculation module is configured to divide the scene data into sub-scenes. For different sub-scenes, the priority of the interaction modality corresponding to the sub-scene is determined based on the frequency of use and interaction performance of the interaction modality in that sub-scene, and an initial interaction priority matrix is ​​obtained. The priority matrix correction module is configured to determine the user's ability adaptability to each interaction modality based on the user's interaction ability feature data, and correct the initial interaction priority matrix based on the ability adaptability. The modality adaptive switching module is configured to acquire current scene data, determine the current sub-scene, and, based on the user interaction priority matrix, determine the highest priority interaction modality in the current sub-scene as the current standby main modality, waiting for user input.

[0012] A third aspect of the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer instructions that, when executed by the processor, cause the electronic device to perform the method described thereon.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0014] A fifth aspect of the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the method described herein.

[0015] The above one or more technical solutions can capture the interaction preferences of different users and whether they possess the capabilities required for different interaction modalities by acquiring user interaction capability feature data and historical interaction data. By constructing an interaction priority matrix for different scenarios and correcting it based on user interaction capability feature data, the interaction modal priority can meet user interaction preferences while conforming to the user's actual interaction capabilities, thereby determining the most suitable main interaction modality in the current scenario and improving the user experience of cockpit interaction. In particular, for disabled users, since the judgment of interaction capability is included in the interaction priority matrix, the priority matrix can prioritize matching the interaction modalities that disabled people can operate, and avoid the modalities that they cannot operate. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 This is a schematic diagram illustrating the working principle of the cockpit multimodal interactive control method in this embodiment of the invention. Figure 2 This is a program module diagram of the cockpit multimodal interaction control device in an embodiment of the present invention. Detailed Implementation

[0018] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0019] In the description of the embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on".

[0020] As described in the background section, current intelligent cockpit multimodal interaction systems rely on a single dominant interaction mode. If this mode cannot effectively recognize user commands, such as in noisy environments where the voice recognition error rate increases significantly, interaction interruptions will occur, affecting user experience. Furthermore, the interaction needs of special groups are not fully considered, and adaptive adaptation based on user interaction capabilities and operating habits is not possible. In particular, no dedicated interaction strategies are designed for disabled users who meet driving qualifications, resulting in a high operating threshold for users, especially those with disabilities, making it difficult to guarantee both interaction experience and driving safety. To address these issues, this invention provides a cockpit multimodal interaction control method. By constructing a personalized interaction priority matrix based on user interaction capabilities and historical interaction data, it achieves adaptive adaptation to scenarios, user preferences, and physiological capabilities, improving the user's in-vehicle interaction experience, especially catering to the needs of disabled users who meet driving qualifications.

[0021] It is understood that the "special population" referred to in this application refers to disabled users who meet the physical requirements for applying for a motor vehicle driver's license under relevant laws and regulations, and who have partial limitations in physiological functions such as limbs, vision, hearing, and speech, but possess normal driving ability. Specifically, this includes, but is not limited to: those with monocular visual impairment, but whose better eye's uncorrected or corrected visual acuity reaches 5.0 or above on the logarithmic visual acuity chart, and whose horizontal visual field reaches 150 degrees; those with hearing impairment, but who, with hearing aids, can distinguish the direction of a sound source at a distance of 50 centimeters from each ear at a tuning fork; and other special user groups who meet the driving qualifications but have differences in interactive abilities. Although these users are qualified to drive, they experience certain operational limitations during cockpit interaction.

[0022] An example environment of one or more embodiments of the present invention includes an in-vehicle controller, a data acquisition module, and an in-vehicle navigation module. Both the data acquisition module and the in-vehicle navigation module are communicatively connected to the in-vehicle controller, which is also connected to a cloud database. Exemplarily, the data acquisition module includes an RGB+D depth camera, an in-vehicle microphone array, an ASR voice recognition engine, a touch sensor, an in-vehicle sensor, and a vibration sensor. The RGB+D depth camera is deployed in an unobstructed position below the rearview mirror in the cabin, facing the front and rear passengers, and is used to collect visual data such as the user's left and right hand movements and fine motor skills, providing data support for judging the user's limb activity and gesture recognition. The in-vehicle microphone array is installed in the center of the cabin ceiling, using a multi-microphone array layout, to accurately collect user voice data, filter in-vehicle ambient noise, and ensure the clarity of the voice data. The ASR voice recognition engine is integrated inside the in-vehicle controller and establishes real-time communication with the in-vehicle microphone array to perform preliminary analysis and evaluation of the collected voice data, extracting voice clarity, etc. Key parameters such as speech rate are included. Touch sensors are integrated into the central control screen, the multi-function area of ​​the steering wheel, and the surface of the rear display screen to collect data such as user touch pressure, touch position coordinates, and operation trajectory, supporting touch input interaction and judging user touch-related abilities. Vehicle sensors include a speed sensor, a light sensor, and a noise sensor, deployed on the wheel hubs, the top of the cabin, and the inside of the central control panel, respectively, to collect scene data such as vehicle speed, interior light intensity, and interior noise decibels. Vibration sensors are integrated into the steering wheel, door armrests, and the edge of the central control screen to collect user response signals to vibration feedback, supporting the implementation of graphic / vibration prompt interaction modes. The vehicle controller is configured to execute the cabin multimodal interaction control method. A cloud database establishes remote communication with the vehicle controller through the vehicle communication module to store user interaction ability characteristic data, historical interaction data, interaction priority matrix, and cabin interaction function library, supporting data retrieval and matrix update data retention.

[0023] Figure 1 A flowchart illustrating an example method for cockpit multimodal interaction control provided in one or more embodiments of the present invention. The method is applied to an in-vehicle controller and includes the following steps: S101. Obtain user interaction capability feature data and historical interaction data, wherein the historical interaction data includes the interaction modality used in each interaction, related scene data, and interaction performance.

[0024] S102. Divide the scene data into sub-scenes. For different sub-scenes, determine the priority of the interaction modal corresponding to the sub-scene based on the frequency of use and interaction performance of the interaction modal in the sub-scene, and obtain the initial interaction priority matrix. S103. Determine the user's ability adaptability to each interaction modality based on the user interaction ability characteristic data, and correct the initial interaction priority matrix based on the ability adaptability. S104. Obtain the current scene data, determine the current sub-scene, and determine the highest priority interaction mode in the current sub-scene as the current standby main mode based on the user interaction priority matrix, and wait for user input.

[0025] By acquiring user interaction capability characteristic data and historical interaction data, it is possible to capture the interaction preferences of different users and whether they possess the capabilities required for different interaction modalities. By constructing an interaction priority matrix for different scenarios and refining it based on user interaction capability characteristic data, the interaction modality priority can satisfy user interaction preferences while aligning with users' actual interaction capabilities. This determines the most suitable primary interaction modality for the current scenario, improving the user experience of cockpit interaction. In particular, for users with disabilities, because the interaction capability assessment is incorporated into the interaction priority matrix, it can prioritize matching interaction modalities that the disabled person can operate, avoiding modalities they cannot operate. Simultaneously, by switching only the highest priority primary interaction modality in the current sub-scenario to a low-power standby state, all modalities do not need to be in a high-power monitoring state simultaneously, effectively reducing system energy consumption.

[0026] In S101, user interaction capability feature data refers to the physiological capabilities a user possesses when performing cockpit interactions, including physical activity, speech expression, visual perception, auditory perception, and reaction speed. It should be noted that because smart cockpit interactions (such as touch, physical button pressing, and gestures) primarily rely on both hands, and the left and right hands have different roles and usage frequencies, it is necessary to separately determine whether the user's left and right hands possess the necessary operational capabilities for cockpit interaction and label the results accordingly. Specifically, physical activity capability is assessed and labeled using visual data such as the user's left and right hand movements, fine motor skill range, and dexterity collected by the in-vehicle RGB+D depth camera. Speech expression capability is evaluated using speech data such as clarity and speed collected by the ASR speech recognition engine; reaction speed is assessed by the time it takes for the user to respond from receiving an interaction prompt. Those skilled in the art will understand that recognizing different body movements based on visual data to determine whether one possesses the corresponding physical activity ability, and judging speech expression ability based on indicators such as speech clarity and speech rate, can both be achieved based on existing machine learning models.

[0027] For example, the user interaction capability feature data includes various physiological capabilities and corresponding levels. Preferably, the levels can be represented by a rating scale. These levels are used to subsequently evaluate whether the user can use the various interaction modalities normally.

[0028] By acquiring user interaction capability characteristic data, we can capture the differences in interaction capabilities among different users, such as differences in operating habits and language expression abilities. This allows us to switch interaction modalities based on the different abilities and preferences of each user. Furthermore, by specifically collecting data on users' physical activity abilities, verbal expression abilities, visual perception abilities, auditory perception abilities, and operational reaction speed, we can capture the boundaries of different users' interaction capabilities. This is also applicable to users with disabilities, facilitating a comprehensive judgment of the required interaction modality based on different user habits, preferences, and operational abilities, thereby improving the user experience.

[0029] Interaction modalities refer to the specific ways in which users interact with the intelligent cockpit system, including but not limited to voice control, touch input, eye-tracking interaction, gesture recognition, physical button backup, and graphic / vibration cues. Voice control allows users to issue cockpit operation commands via voice; touch input allows users to input control commands by touching or gesturing on touch interfaces such as the central control screen, rear-seat displays, and steering wheel; eye-tracking interaction allows users to browse and select cockpit interface options by looking at them, adapting to users with limited physical mobility; gesture recognition uses preset, simplified gestures to quickly trigger functions; physical button backup input allows users to press physical buttons on easily accessible locations such as the steering wheel and door armrests; and graphic / vibration cues deliver cockpit interaction information by outputting high-contrast graphics and triggering vibration feedback for users with hearing or speech impairments.

[0030] Historical interaction data includes scene data such as vehicle driving status (moving, parking, stationary), vehicle speed, in-vehicle environment (light intensity, noise level), and road condition information, which are collected through vehicle sensors, vehicle infotainment system, and navigation module.

[0031] Interaction performance data includes metrics such as recognition accuracy, response speed, and misoperation rate for each modality, collected through the execution modules and recognition engine of each interaction modality. Recognition accuracy is the precision with which the interaction modality recognizes user input commands. It is calculated as the percentage of successfully recognized interactions for that modality within a sub-scenario out of the total number of interactions for that modality within that sub-scenario. Successful recognition is determined by the absence of secondary correction by the user. Response speed is the time taken for the interaction modality to output a recognition result / activate the function from receiving user input. It can be represented by the average response time of multiple interactions for that modality within a sub-scenario. Misoperation rate is the percentage of invalid operations for that interaction modality within a sub-scenario. Invalid operations are identified by secondary correction or misoperation by the user. Misoperations include accidental touches, ambiguous input signals, and operation intentions exceeding the scope of the cockpit's executable functions. For example, an operation that meets the trigger threshold of the interaction modality but is determined to be invalid input after interpretation of the operation intention and matching with the cockpit interaction function is recorded as a misoperation. The specific judgment process is detailed in step S105 below.

[0032] The above data collection and index calculation are consistent with the logic of historical interaction data collection in S101 and interaction performance evaluation in S102. The calculation results are used for comprehensive scoring and priority ranking of each modality to ensure the scientificity and feasibility of the initial interaction priority matrix.

[0033] The user feature data and historical interaction data collected above are preprocessed in a unified manner to remove outliers, fill in missing values, quantize and encode unstructured data, standardize all data into a unified format, and associate them with unique user IDs, scene codes, and modal codes to form a structured data set.

[0034] In step S102, noise and light intensity parameters are extracted from the scene data and classified into levels, such as 1-5, with higher levels indicating stronger environmental interference, used to quantify the level of environmental interference. Simultaneously, vehicle driving status and speed parameters from the scene data are combined to determine the driving safety level for each scenario, for example, divided into high, medium, and low levels. High-speed driving corresponds to a high safety level, low-speed urban driving corresponds to a medium safety level, and parking and stationary states correspond to a low safety level. Based on the combination of environmental interference level and driving safety level, sub-scenes are divided, with different level combinations corresponding to different sub-scenes. Then, historical interaction modalities, associated scene data, and interaction performance are retrieved from historical interaction data, and the historical interaction modalities and performance are matched to the corresponding sub-scenes.

[0035] Statistical analysis is performed on the historical interaction modalities in each sub-scenario. The total number of historical uses of each interaction modality in that sub-scenario is counted, and the usage frequency percentage of each modality is calculated. Simultaneously, the interaction performance of each interaction modality in that sub-scenario is extracted, and the interaction performance is evaluated based on the interaction performance. A comprehensive evaluation is then performed on each modality based on its usage frequency and interaction performance evaluation results. The comprehensive evaluation result is used to determine the ranking priority. For example, interaction performance indicators may include recognition accuracy, response speed, and error rate. Each indicator is normalized and then weighted to obtain the performance evaluation result of that interaction modality. Finally, the usage frequency percentage and interaction performance evaluation results are normalized separately and then weighted and summed to obtain a comprehensive score. Interaction modalities are ranked from highest to lowest based on their comprehensive scores, with higher scores indicating higher priority. If comprehensive scores are the same, the modality with the higher comprehensive interaction performance score is given priority.

[0036] Finally, using sub-scenes as indices, the interaction modalities are sorted according to priority to construct an initial interaction priority matrix. For example, for visually impaired users, the priority is set as voice > vibration prompts > physical buttons.

[0037] In step S103, a mapping relationship is established between interaction modalities and required physiological ability levels. For example, voice control requires meeting standards for both voice expression and auditory perception, while touch input requires meeting standards for both limb movement and visual perception. Based on this mapping relationship, user interaction ability characteristic data is analyzed to determine the user's ability adaptability to each interaction modality. The suitability is then used to determine the rationality of the priority ranking of each interaction modality in the initial interaction priority matrix. If the priority of a certain interaction modality is not compatible with the user's corresponding physiological ability, the priority of that modality is lowered, while the priority of modalities compatible with the user's physiological ability is raised, ensuring that the corrected matrix accurately reflects the user's actual interaction ability.

[0038] For example, the user's physiological ability ratings in S101 are compared with the minimum physiological ability thresholds required for each interaction modality to calculate the fit score. For example, fit score = (user's physiological ability rating ÷ minimum physiological ability threshold required for that modality) × 100%. Fit score thresholds are set: fit score ≥ 80%, considered fully fit, priority remains unchanged or is appropriately increased; 50% ≤ fit score < 80%, considered partially fit, priority is reduced by 1-2 levels; fit score < 50%, considered completely unfit, priority is reduced to the lowest level (below all fit modalities). For example, if the user's limb movement ability rating is 5 points, and the minimum limb movement ability threshold required for touch input is 6 points, the fit score is approximately 83.3%, considered fully fit, and the touch input priority remains unchanged; if the user's voice expression ability rating is 4 points, and the minimum voice expression ability threshold required for voice control is 6 points, the fit score is approximately 66.7%, considered partially fit, and the voice control priority is reduced by 1 level.

[0039] In S104, the vehicle sensors, vehicle system and navigation module are called in real time to collect current vehicle status data and environmental data. The vehicle status data includes the current vehicle driving status and real-time vehicle speed, and the environmental data includes the current in-vehicle light intensity and noise decibels. According to the sub-scene division rules in S102, the corresponding combination of environmental interference level and driving safety level is matched to determine the current sub-scene.

[0040] The interaction priority matrix is ​​invoked to locate the interaction modality priority sequence within the sub-scene, and the interaction modality with the highest priority in the sequence is selected as the current primary interaction modality. The determined primary interaction modality is switched to a low-power standby state, and the corresponding input monitoring module is activated to monitor user input signals in real time. Simultaneously, the user is notified of the current interaction modality via voice, the central control screen, or other means. The primary interaction modality monitoring module continuously waits for user input. If a user input signal is detected, it immediately performs preliminary preprocessing and transmits it to the corresponding modality's recognition engine to prepare for subsequent command parsing and response. If no user input is detected, the primary interaction modality will remain in standby mode until an input signal is detected or a scene change or modality switching condition is triggered.

[0041] In some embodiments, the method further includes step S105: real-time monitoring of input signals and scene data, and triggering interaction mode switching when the input signal modality does not match the current interaction modality, or when the sub-scene changes. This step corresponds to two interaction mode switching scenarios.

[0042] In actual cockpit interaction, users may initiate input commands outside the current primary interaction mode due to operating habits, immediate needs, or insufficient adaptability of the primary interaction mode. To address this, some embodiments switch the primary interaction mode to a low-power standby state while simultaneously setting other interaction modes as backup modes in a background ready state, without activating active monitoring but maintaining rapid wake-up capability. For example, low-power trigger thresholds are preset for all interaction modes, such as touch pressure not less than 5N, button press travel not less than 2mm, and gesture matching preset contours. When an interaction mode is set as the primary interaction mode, other backup interaction modes sense the user's active input based on this threshold. In the low-power standby state of the primary interaction mode, only the core monitoring unit is activated, reducing unnecessary power consumption while ensuring response latency is controlled within 500ms, balancing power consumption and interaction smoothness. For users with disabilities, once the main interaction mode is activated, adaptive prompts will be automatically triggered, such as voice prompts for visually impaired users and vibration prompts for hearing impaired users, informing them of the current main interaction mode and operation method, thus preventing operational errors due to users' lack of understanding of the current interaction mode. Simultaneously, the monitoring module will capture subtle user input movements in real time. Considering the limited range of motion and input delays experienced by users with physical disabilities, the monitoring sensitivity is optimized to ensure that no input signal is missed, further improving the ease of interaction for users with disabilities.

[0043] Based on this, when one interaction mode is set as the primary interaction mode, other backup interaction modes sense user input actions based on trigger thresholds. When an input action that meets the trigger threshold of a backup interaction mode is detected, that backup interaction mode is activated, the user input signal is parsed, and the operation intent is analyzed. If the operation intent is successfully interpreted and matches the cockpit interaction function, the input signal is considered valid, converted into a cockpit control command, and executed, while the standby monitoring of the primary interaction mode is paused. For example, a successfully interpreted operation intent must include the operation object (such as cockpit functions like navigation, volume, air conditioning, and windows) and the operation command (such as opening, closing, adjusting, or switching). If the operation object and operation command can be clearly extracted, the operation intent is considered successfully interpreted. The system has a preset cockpit interaction function library, which contains all executable cockpit interaction functions and their corresponding identifiable operation intent sets. The interpreted operation intent is compared with the intent sets corresponding to each interaction function in the function library. If a completely matching cockpit interaction function exists, the operation intent is considered to match the cockpit interaction function. The input signal is considered valid only if both of the above steps are successful, and is then converted into a cockpit control command and executed. If either step fails, the input signal is considered invalid, and the system immediately terminates the wake-up state of the backup interaction mode and restores the main interaction mode to standby state.

[0044] Based on the aforementioned background readiness settings and low-power wake-up mechanism for backup interaction modes, the system can flexibly switch interaction modes to cope with different user input scenarios and environmental changes. Through this judgment process, false wake-ups of modes caused by invalid input and waste of system resources are avoided. Simultaneously, the validity and accuracy of user input in non-primary interaction modes are ensured, guaranteeing the continuity and reliability of the interaction process. This adapts to the operating habits and immediate needs of different users, improving the flexibility and user experience of cockpit interaction, especially catering to the personalized interaction scenarios of disabled users, and reducing the operational threshold and probability of errors in non-primary interaction mode input.

[0045] In some embodiments, after the backup interaction mode is awakened and executes control commands, the system continuously monitors the interaction status of the backup mode. If the user does not initiate input commands for that mode within a preset time, the system will automatically switch back to the original primary interaction mode and restore its low-power standby monitoring state, ensuring the continuity and rationality of the interaction. Simultaneously, to address potential input delays and discontinuous operations for special groups, the duration after the backup mode is awakened can be flexibly adjusted. For example, the error tolerance time for interpreting operational intent can be optimized for users with poor language skills, avoiding interaction interruptions due to operational delays. Furthermore, the system records the scenario, triggering reason, and interaction result of each mode switch, incorporating them into historical interaction data. This provides data support for subsequent updates to the interaction priority matrix, further optimizing the mode switching strategy and making the interaction adaptation more aligned with users' long-term operating habits. Especially for the fixed operating preferences of disabled users, personalized adaptation of mode switching is gradually achieved, minimizing the interaction difficulty for disabled users and ensuring that all types of users can conveniently and safely complete cockpit interaction operations.

[0046] In addition, real-time data collection of the current scene, including vehicle status data and environmental data, is used to determine the current sub-scene and compare it with the sub-scene corresponding to the current main interaction modality. If they are inconsistent, the sub-scene is considered to have changed. For example, a vehicle switching from a medium safety level and low noise interference scenario (corresponding to low-speed driving in urban areas) to a high safety level and medium noise interference scenario (corresponding to high-speed driving), or a sudden increase in noise inside the vehicle causing the environmental interference level to rise from level 2 to level 5, all fall under the category of a sub-scene change. If a sub-scene change is determined, the interaction priority matrix is ​​invoked to locate the interaction modality priority sequence corresponding to the new sub-scene. The interaction modality with the highest priority in this sequence is selected as the main interaction modality under the new sub-scene. The newly determined main interaction modality is switched to a low-power standby state, its corresponding input monitoring module is activated, and the remaining interaction modalities are updated to the backup interaction modalities under the new sub-scene and set to the background ready state.

[0047] In some embodiments, steps S101-S103 are repeated at set time intervals or in response to user requests to update the interaction priority matrix. This allows for real-time adaptation to changes in user interaction capabilities, iterations in user operating habits, and the accumulation of historical interaction data. It ensures that the interaction priority matrix matches the user's actual needs and the cockpit interaction scenario, further enhancing the personalization and reliability of the system interaction.

[0048] One or more embodiments of the present invention also provide a cockpit multimodal interaction control device, comprising: a historical data acquisition module 201, configured to acquire user interaction capability feature data and historical interaction data, wherein the historical interaction data includes the interaction modality used in each interaction, associated scene data, and interaction performance; a priority matrix calculation module 202, configured to divide sub-scenes according to scene data, and for different sub-scenes, determine the priority of the interaction modality corresponding to the sub-scene according to the frequency of use and interaction performance of the interaction modality in the sub-scene, thereby obtaining an initial interaction priority matrix; a priority matrix correction module 203, configured to determine the user's ability adaptability to each interaction modality according to the user interaction capability feature data, and correct the initial interaction priority matrix according to the ability adaptability; and a modality adaptive switching module 204, configured to acquire current scene data, determine the current sub-scene, and determine the interaction modality with the highest priority in the current sub-scene as the current standby main modality, waiting for user input, according to the user interaction priority matrix.

[0049] One or more embodiments of the present invention also provide an electronic device that can be used to implement the cockpit multimodal interactive control method in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0050] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage. The computer program may be stored in the ROM. When the processor executes the computer program, it implements the above-described cockpit multimodal interactive control method.

[0051] In some embodiments, the program may be tangibly contained in a computer-readable medium, which may include a device (such as a memory) or other storage device accessible by the device. The program may be loaded from the computer-readable medium into RAM for execution. The computer-readable medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, where a computer program is stored that, when executed by a processor, implements the aforementioned cockpit multimodal interactive control method.

[0052] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a server or terminal, they generate, in whole or in part, the processes or functions described in the embodiments of this application. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to the server or terminal, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, and magnetic tape), an optical medium (e.g., digital video disk (DVD), etc.), or a semiconductor medium (e.g., solid-state drive).

[0053] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0054] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A cockpit multimodal interactive control method, characterized in that, Includes the following steps: Acquire user interaction capability feature data and historical interaction data, wherein the historical interaction data includes the interaction modality used in each interaction, related scene data, and interaction performance; The scene data is divided into sub-scenes. For different sub-scenes, the priority of the interaction modality corresponding to the sub-scene is determined based on the frequency of use and interaction performance of the interaction modality in the sub-scene, and an initial interaction priority matrix is ​​obtained. The user's ability to adapt to each interaction modality is determined based on the user's interaction ability characteristic data, and the initial interaction priority matrix is ​​corrected based on the ability adaptability. Obtain current scene data, determine the current sub-scene, and based on the user interaction priority matrix, determine the highest priority interaction mode in the current sub-scene as the current standby main mode, waiting for user input.

2. The cockpit multimodal interactive control method as described in claim 1, characterized in that, The user interaction capability feature data includes the user's physiological ability and capability level to adapt to cockpit interaction operations.

3. The cockpit multimodal interactive control method as described in claim 1, characterized in that, Based on the usage frequency and interaction performance of the interaction modal in the sub-scenario, the priority of the interaction modal corresponding to the sub-scenario is determined by: for the historical interaction modal in each sub-scenario, counting the total number of historical uses of each interaction modal in the sub-scenario and calculating the usage frequency ratio of each modal; at the same time, evaluating the interaction performance of each interaction modal in the sub-scenario; and determining the ranking priority based on the evaluation results of the usage frequency and interaction performance of each modal.

4. The cockpit multimodal interactive control method as described in claim 1 or 2, characterized in that, The modification of the initial interaction priority matrix based on capability adaptability includes: the mapping relationship between interaction modalities and required physiological ability levels, determining the user's capability adaptability to each interaction modality; and verifying the ranking of each interaction modality in the initial interaction priority matrix based on capability adaptability.

5. The cockpit multimodal interactive control method as described in claim 1, characterized in that, In addition to the primary mode, all other backup interaction modes are equipped with low-power trigger thresholds. When an input action that meets the trigger threshold of a backup interaction mode is detected, the backup interaction mode is woken up, and the operation intention in the user input signal is parsed. If the operation intention is successfully interpreted and can be matched with the cockpit interaction function, the input signal is considered valid, converted into a cockpit control command and executed, and the standby monitoring of the primary interaction mode is paused.

6. The cockpit multimodal interactive control method as described in claim 1, characterized in that, The method further includes: while waiting for user input, monitoring scene data in real time to determine whether the sub-scene has changed; if so, triggering an interaction mode switch; and determining the interaction mode with the highest priority in the current sub-scene as the current standby main mode based on the user interaction priority matrix.

7. A cockpit multimodal interactive control device, characterized in that, include: The historical data acquisition module is configured to acquire user interaction capability feature data and historical interaction data, wherein the historical interaction data includes the interaction modality used in each interaction, related scene data, and interaction performance. The priority matrix calculation module is configured to divide the scene data into sub-scenes. For different sub-scenes, the priority of the interaction modality corresponding to the sub-scene is determined based on the frequency of use and interaction performance of the interaction modality in that sub-scene, and an initial interaction priority matrix is ​​obtained. The priority matrix correction module is configured to determine the user's ability adaptability to each interaction modality based on the user's interaction ability feature data, and correct the initial interaction priority matrix based on the ability adaptability. The modality adaptive switching module is configured to acquire current scene data, determine the current sub-scene, and, based on the user interaction priority matrix, determine the highest priority interaction modality in the current sub-scene as the current standby main modality, waiting for user input.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images