Vehicle cockpit control methods, electronic equipment and vehicle

By detecting user interaction input through sensors in the vehicle cabin, the audio system is controlled to generate audio signals in real time, solving the problem of limited entertainment functions in existing technologies, enabling rich audio interactive creation and intelligent enhancement, and improving the driving experience.

CN123061301APending Publication Date: 2026-11-10BYD CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611134618.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-28
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

The entertainment functions of existing vehicles are limited to playing preset audio content, which cannot meet the diverse needs of passengers, resulting in a monotonous auditory experience and affecting the driving and riding experience.

Method used

By detecting user interaction input through sensors in the vehicle cabin, the audio system is controlled to generate audio signals in real time, supporting drivers and passengers to actively create audio interactive content and achieving real-time response between audio output and interactive input.

Benefits of technology

It enriches the forms of in-cabin entertainment, enhances the fun of in-cabin interaction and the level of vehicle intelligence, and improves the experience of drivers and passengers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123061301A_ABST
    Figure CN123061301A_ABST
Patent Text Reader

Abstract

This application provides a vehicle cockpit control method, electronic device, and vehicle, applicable to the field of vehicle technology. It aims to enhance the interactive fun of the cockpit and the vehicle's intelligence level, thereby improving the experience for drivers and passengers. The method includes: when a target function is activated, responding to user input, controlling the vehicle's audio system to output audio signals.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a control method for a vehicle cockpit, electronic equipment, and a vehicle. Background Technology

[0002] With the continuous development of vehicle technology, vehicles are now not only a means of transportation, but also a mobile intelligent space that can accommodate various scenarios such as rest, work, and entertainment, meeting the growing and diverse driving and riding needs of passengers.

[0003] Among them, hearing is one of the senses through which passengers perceive and interact. The quality of the auditory experience directly affects the passenger's riding experience and driving experience. In related technologies, the vehicle's entertainment function is limited to playing preset audio content, which is a single function and cannot meet the needs of passengers. Summary of the Invention

[0004] The purpose of this application is to provide a vehicle cockpit control method, electronic equipment, and vehicle, which aims to enhance the interactive fun of the cockpit and the level of vehicle intelligence, thereby improving the experience of drivers and passengers.

[0005] In a first aspect, a method for controlling a vehicle cockpit is provided, comprising: when a target function is activated, controlling the vehicle's audio system to output an audio signal in response to user interaction input.

[0006] Beneficial effects: When the target function is activated, by converting the user's interactive input into audio signals for the audio device, audio signals are generated based on the user's real-time operation. Compared with the existing technology that can only play audio, this application breaks away from the limitation of one-way intelligent audio playback through real-time response between interactive input and audio output. It supports drivers and passengers to actively create audio interactive content, enriches the form of in-cabin entertainment, enhances the fun of in-cabin interaction and the level of vehicle intelligence, thereby improving the experience of drivers and passengers.

[0007] Optionally, the interactive input includes: a touch action; in response to the user's interactive input, controlling the vehicle's audio system to output an audio signal includes: in response to the user's touch action on a component of the vehicle's cabin, controlling the vehicle's audio system to output an audio signal.

[0008] Optionally, the vehicle includes multiple cabin components; each cabin component includes multiple interaction areas; wherein, one cabin component corresponds to one musical instrument; and the interaction area of ​​one cabin component corresponds to a part of one musical instrument.

[0009] Optionally, when the cockpit component is a steering wheel, controlling the vehicle's audio system to output an audio signal includes: determining the target area on the steering wheel where the touch action is applied; the target area is one or more of a plurality of interactive areas on the steering wheel; generating a corresponding audio signal based on a target note or target chord corresponding to the target area, and controlling the audio system to output the audio signal; wherein, one interactive area corresponds to one note or one chord.

[0010] Optionally, the touch action includes multiple actions, wherein some of the touch actions are triggered simultaneously or sequentially; controlling the vehicle's audio device to output an audio signal includes: determining the cabin components corresponding to the multiple touch actions respectively; generating an audio signal based on the type of instrument corresponding to each touch action, and controlling the audio device to output the audio signal.

[0011] Optionally, one interactive area corresponds to one musical note; in response to a user's touch action on a component of the vehicle's cabin, the system controls the vehicle's audio device to output an audio signal, including: determining the score to be played, the score to be played including: multiple notes arranged in sequence; based on the score to be played, controlling the vehicle's display device and / or lighting device to output a guidance instruction, the guidance instruction being used to prompt the user to perform a touch action in the corresponding interactive area; and based on the touch action performed by the user, generating an audio signal and controlling the audio device to output it.

[0012] Optionally, in response to user interaction input, controlling the vehicle's audio device to output an audio signal includes: in response to user interaction input, controlling the vehicle's audio device to output an audio signal, and controlling at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device.

[0013] Optionally, controlling the vehicle's audio system to output an audio signal and controlling at least one of the lighting device, display device, and haptic feedback device to link with the audio system includes: determining the light emission parameters of the lighting device based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input; the light emission parameters include at least: brightness, color, flashing frequency, or lighting sequence; and controlling the lighting device to link with the audio system based on the light emission parameters.

[0014] Optionally, controlling the vehicle's audio system to output an audio signal and controlling at least one of the lighting device, display device, and haptic feedback device to link with the audio system includes: controlling the display parameters of the display device based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input, wherein the display parameters include at least: color, brightness, display position, or display content; and controlling the display device to link with the audio system based on the display parameters.

[0015] Optionally, controlling the vehicle's audio system to output an audio signal and controlling at least one of the lighting device, display device, and haptic feedback device to link with the audio system includes: controlling the haptic parameters of the haptic feedback device based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input, wherein the haptic parameters include at least: vibration intensity, vibration frequency, or vibration timing; and controlling the haptic feedback device to link with the audio system based on the haptic parameters.

[0016] Optionally, in response to user interaction input, controlling the vehicle's audio device to output an audio signal includes: determining audio parameters of the audio to be generated based on interaction parameters of the interaction input; the audio parameters are used to characterize at least one of the following: note, timbre, volume, beat, sound image position, and sound effect of the audio to be generated; and arranging and generating an audio signal based on the audio parameters of the audio to be generated and outputting it.

[0017] Optionally, when the interactive input is a touch action, the interactive parameters are touch parameters; the touch parameters include at least one of the following: the position, force, duration, and speed of the touch action; the timbre and / or note is determined based on the interactive area corresponding to the touch action; the volume is determined based on the force of the touch action; the beat is determined based on the frequency of the touch action; the sound image position is determined based on the position of the cockpit component corresponding to the touch action in the vehicle cockpit; the sound image position is used to characterize the position of the audio to be generated in the stereo field.

[0018] Optionally, the touch action includes multiple actions; in response to the user's touch action on the vehicle's cabin components, the system controls the vehicle's audio device to output an audio signal, including: determining the trigger time of the touch action; for multiple touch actions at the same time, synthesizing the sub-audio signals corresponding to each touch action to generate an audio signal and outputting it.

[0019] Optionally, the cockpit components corresponding to the touch action include multiple components; in response to the user's touch action on the cockpit components of the vehicle, the audio device of the vehicle is controlled to output an audio signal, including: determining the sub-audio signal corresponding to each cockpit component based on the touch action corresponding to each cockpit component; synthesizing the sub-audio signals to obtain an audio signal and outputting it.

[0020] Optionally, based on the touch action corresponding to each cockpit component, the sub-audio signal corresponding to each cockpit component is determined, including: determining the instrument type corresponding to each cockpit component; and determining the sub-audio signal corresponding to the touch action based on the instrument type, so that multiple instruments output audio signals based on the same beat.

[0021] Optionally, the cockpit components corresponding to the touch action include multiple components; in response to the user's touch action on the cockpit components of the vehicle, the audio device of the vehicle is controlled to output an audio signal, including: adjusting the volume parameters of the sub-audio signals corresponding to each cockpit component based on the number of touch actions corresponding to each cockpit component within a preset time period, so as to achieve volume balance.

[0022] Optionally, the cockpit components corresponding to the touch action include multiple components; in response to the user's touch action on the cockpit components of the vehicle, the audio device of the vehicle is controlled to output an audio signal, including: determining the corresponding timbre parameters, reverberation parameters and / or delay parameters based on the target music style; generating an audio signal and outputting it according to the timbre parameters, reverberation parameters and / or delay parameters.

[0023] Optionally, in response to a user's touch action on a component of the vehicle's cabin, the audio device of the vehicle is controlled to output an audio signal, including: if the difference between the output time of the guidance command and the trigger time of the touch action is less than a first preset duration threshold, the audio device is controlled to output a note corresponding to the guidance command, and the lighting device is controlled to output light feedback.

[0024] Optionally, in response to a user's touch action on a component of the vehicle's cabin, controlling the vehicle's audio device to output an audio signal further includes: if the difference between the trigger time of the touch action and the output time of the guidance command is greater than or equal to a first preset duration threshold and less than a second preset duration threshold, correcting the output time of the audio signal corresponding to the touch action to a preset time, wherein the preset time is the closest musical score beat point to the trigger time.

[0025] Optionally, in response to a user's touch action on a cockpit component of the vehicle, the audio device of the vehicle is controlled to output an audio signal, including: if the actual interaction area corresponding to the touch action does not match the target interaction area corresponding to the guidance instruction, the incorrect note corresponding to the actual interaction area is replaced with the target note corresponding to the target interaction area, and / or, a transition sound effect is output to replace the incorrect note.

[0026] Optionally, in response to a user's touch action on a component of the vehicle's cabin, the system controls the vehicle's audio device to output an audio signal, including: if the number of consecutive correct taps by the user exceeds a preset number, performing at least one of the following operations: outputting a continuous tap sound effect; controlling the vehicle's lighting device to adjust the color temperature of the lights; and controlling the audio device to add drum beats and / or bass lines to the audio signal.

[0027] Optionally, in response to a user's touch action on a cabin component of the vehicle, controlling the vehicle's audio device to output an audio signal includes: responding to a sequence of touch actions by the user on a cabin component of the vehicle; determining a note sequence corresponding to the touch action sequence based on the touch action sequence; and controlling the vehicle's audio device to output an audio signal based on the note sequence.

[0028] Optionally, in response to user interaction input, controlling the vehicle's audio system to output an audio signal includes: acquiring an interaction signal generated by the interaction input through an onboard sensor; extracting features from the interaction signal to determine interaction features; using the interaction features to characterize the signal quality of the interaction signal; determining the interaction signal as a valid interaction signal if the interaction feature characterization meets preset quality conditions; and controlling the audio system in the vehicle to output an audio signal based on the valid interaction signal.

[0029] Optionally, the interactive signals may be acquired through at least one of the following methods: via a sensor corresponding to the steering wheel; via a sensor corresponding to the inner panel and / or dashboard; via a sensor corresponding to the armrest and / or seat and / or carpet; via a sensor corresponding to the brake pedal; via a camera; or via a microphone.

[0030] Optionally, the interactive input is a touch action; the interactive signal is a tapping signal; when the cockpit component is a steering wheel, the valid tapping signal is determined in the following way: based on a first condition, the tapping signals obtained by the sensors corresponding to the steering wheel are filtered to obtain candidate tapping signals; the first condition includes: the amplitude of the tapping signal is greater than a preset amplitude threshold, the rate of change of the amplitude of the tapping signal is greater than a preset rate threshold, and the duration of the tapping signal is less than a preset tapping time; based on a second condition, the candidate tapping signals are filtered to obtain valid tapping signals; the second condition includes: the target grip parameter is less than or equal to a preset grip parameter threshold; the target grip parameter includes: grip duration and / or grip contact area; the time difference between the triggering time of the tapping signal and the triggering time of the steering wheel button signal is less than or equal to a first time difference; the time difference between the triggering time of the tapping signal and the occurrence time of the vehicle vibration signal is less than or equal to a second time difference.

[0031] Optionally, the brake pedal corresponds to a first transmission link and a second transmission link; the first transmission link and the second transmission link are isolated from each other; the first transmission link is used to transmit braking signals; and the second transmission link is used to transmit interactive signals.

[0032] Optionally, the input signal of the first transmission link includes the sensor acquisition signal of the brake pedal, and the interaction signal transmitted by the second transmission link is a copy of the acquisition signal in the first transmission link.

[0033] Optionally, the steering wheel corresponds to a third transmission link and a fourth transmission link; the third transmission link and the fourth transmission link are isolated from each other; the signal transmitted by the third transmission link is used for driver grip detection; the fourth transmission link is used to acquire and transmit interactive signals.

[0034] Optionally, the input signal of the third transmission link includes the acquisition signal of the sensor corresponding to the steering wheel, and the interaction signal transmitted by the fourth transmission link is a copy of the acquisition signal in the third transmission link.

[0035] Optionally, in response to user interaction input, controlling the vehicle's audio system to output an audio signal includes: when the vehicle meets preset safety conditions, controlling the vehicle's audio system to output an audio signal in response to user interaction input; the preset safety conditions include: the vehicle is in P gear; the vehicle speed is zero; the vehicle is in parking brake mode; the vehicle's braking control function is not activated; the vehicle's steering control function is not in emergency takeover mode; and the vehicle's active safety functions are not activated.

[0036] Optionally, the above method further includes: when the vehicle does not meet the preset safety conditions and the source of the interactive input is the driver's position, the second transmission link corresponding to the brake pedal does not transmit the initial signal, and / or the fourth transmission link corresponding to the steering wheel does not transmit the initial signal, and / or the target function is downgraded or turned off.

[0037] Optionally, the above method further includes: when the vehicle does not meet the preset safety conditions, the source of the interactive input is a non-driver's position, and the touch action is applied to the steering wheel, when the vehicle speed is less than or equal to the preset vehicle speed, the control target function operates in a restricted mode; or when the vehicle speed is greater than the preset vehicle speed, the control target function is downgraded or turned off.

[0038] Optionally, the cabin components include at least one of the following: steering wheel, interior panel, seat, dashboard, armrest, carpet, and brake pedal.

[0039] Optionally, interactive input may include at least one of the following: tapping, slapping, light touching, heavy tapping, continuous rhythmic movements, stomping movements, voice, gestures, or swipes.

[0040] Optionally, the target function is a function mode that generates corresponding audio signals based on user interactive input.

[0041] Optionally, the target function includes at least one of the following modes: performance mode; performance mode is used to instruct the mapping of cockpit components and / or interactive areas to different notes to simulate instrument playing; ensemble mode; ensemble mode is used to instruct the mapping of different cockpit components to different instrument types to simulate multi-instrument ensemble playing; sheet music guidance mode; sheet music guidance mode is used to instruct and guide the user to perform different interactive inputs to play sheet music.

[0042] In a second aspect, an electronic device is also provided, comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional vehicle cockpit control methods described in the first aspect above.

[0043] Thirdly, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a device, the device is able to perform any of the optional vehicle cockpit control methods described in the first aspect.

[0044] Fourthly, a vehicle is also provided, comprising: the electronic equipment of the second aspect above, or the computer-readable storage medium of the third aspect.

[0045] Fifthly, a computer program product is also provided, comprising computer instructions that, when executed on a processor of a device, enable the device to perform any of the optional vehicle cockpit control methods described in the first aspect above. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A structural schematic diagram (I) of a vehicle cockpit system provided for an embodiment of this application; Figure 2 A structural schematic diagram (II) of a vehicle cockpit system provided for an embodiment of this application; Figure 3 This is a schematic diagram showing the location of the tappable points provided in an embodiment of this application; Figure 4 A communication architecture diagram of a vehicle cockpit system provided in this application embodiment; Figure 5 A linkage architecture diagram of a vehicle cockpit system provided in this application embodiment; Figure 6 A schematic flowchart illustrating a vehicle cockpit control method provided in an embodiment of this application; Figure 7 A schematic diagram of a vehicle cockpit interaction mode provided in an embodiment of this application; Figure 8 A schematic diagram illustrating an effective identification process for a tapping signal, provided as an embodiment of this application; Figure 9 A schematic diagram of a dual-link design for a brake pedal provided in an embodiment of this application; Figure 10 A schematic diagram of a dual-link design for a steering wheel provided in an embodiment of this application; Figure 11 A schematic diagram illustrating a vehicle cockpit system safety gate control as provided in this application embodiment; Figure 12 A schematic diagram of an architecture for music interaction based on a steering wheel, provided for an embodiment of this application; Figure 13 A schematic diagram illustrating a music interaction based on a steering wheel, provided as an embodiment of this application; Figure 14 A schematic diagram of gating logic for music interaction based on a steering wheel, provided for an embodiment of this application; Figure 15 A schematic diagram of the structure of a vehicle cockpit control device provided in an embodiment of this application; Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] In the embodiments of this application, the terms "first," "second," "third," "fourth," "fifth," and "sixth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," "fourth," "fifth," and "sixth" may explicitly or implicitly include one or more of that feature.

[0049] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0050] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0051] With the continuous development of vehicle technology, vehicles are now not only a means of transportation, but also a mobile intelligent space that can accommodate various scenarios such as rest, work, and entertainment, meeting the growing and diverse driving and riding needs of passengers.

[0052] Among them, hearing is one of the senses through which passengers perceive and interact. The quality of the auditory experience directly affects the passenger's riding experience and driving experience. In related technologies, the vehicle's entertainment function is limited to playing preset audio content, which is a single function and cannot meet the needs of passengers.

[0053] Based on this, this application provides a vehicle cockpit control method, electronic device, and vehicle. When the target function is activated, the user's interactive input is converted into an audio signal for the audio device, thereby generating an audio signal based on the user's real-time operation. Compared with the prior art, which can only play audio, this application overcomes the limitation of one-way intelligent audio playback through real-time response between interactive input and audio output. It supports drivers and passengers to actively create audio interactive content, enriches the form of cockpit entertainment, enhances the fun of cockpit interaction and the level of vehicle intelligence, and thus improves the experience of drivers and passengers.

[0054] In some embodiments, such as Figure 1 As shown, this application provides a vehicle cockpit system, including: a sensor 10, a controller 20, and an audio device 30.

[0055] Sensor 10 is used to detect user interaction input, and controller 20 is used to convert the user interaction input into an audio signal and control the audio signal to be output in speaker device 30.

[0056] like Figure 2 As shown, sensors may include: hands-off detection (HOD) / vibration sensors, door panel piezoelectric pads, inertial measurement units (IMUs), pressure arrays, pedal displacement or pressure sensors, driver monitoring system (DMS) cameras, microphones, etc., to detect user interaction input.

[0057] The controller can be any algorithm unit with control functions within the vehicle, such as a central controller or a domain controller. In this application, since the control is of the cockpit components, the controller of this application can be a cockpit domain controller.

[0058] In some embodiments, the vehicle cockpit system of this application may further include: a display device, a lighting device, and a haptic feedback device.

[0059] As one possible implementation, such as Figure 2 As shown, the cockpit domain controller (or interactive orchestration controller) reads vehicle speed, gear position, parking brake, brake activation status, driving task priority, and vehicle safety control status. It first identifies the tapping event and then performs safety gating. When preset safety conditions are met, the system maps the tapping event to a music event and uses a unified beat and timestamp for multimodal linkage (sound / light / screen / touch), i.e., orchestrating audio, ambient lighting, vehicle display, and haptic feedback, and performing music mapping generation processing to output data such as notes, timbres, and Musical Instrument Digital Interface (MIDI). Finally, the cockpit domain controller sends control commands to the audio amplifier (speakers, notes and spatial sound images), ambient lighting controller (color / brightness / flow), vehicle display system (scoreboard / animation / area highlighting), and haptic feedback actuators (seat / steering wheel / armrest) to achieve multimodal interaction. Simultaneously, it interacts with the vehicle status gateway to synchronously acquire vehicle operating information such as speed, gear, and braking, completing the entire closed-loop control. When safety conditions are not met, the system disables music input from the steering wheel and pedals, or retains only low-interference light cues and passive music rhythms.

[0060] Specifically, users can activate the cabin instrument mode via the in-vehicle infotainment system, voice commands, customizable buttons on the steering wheel, the rear screen, or through camping / charging waiting scenarios. The vehicle status gating module reads vehicle speed, gear position, parking brake status, braking status, and safety control status to determine whether entry into the full music mode is permitted. The multi-area tapping perception module collects tapping signals from the steering wheel, door panels, dashboard, armrests, seats, carpets, or pedals, and assigns a unified timestamp to each signal. The tapping event recognition module filters, extracts features, and fuses multiple sources to output standardized tapping events. The music mapping generation module generates music events based on the tapping area, intensity, rhythm, user-selected instrument mode, and current score status. The multimodal linkage orchestration module synchronously generates audio, ambient lighting, in-vehicle display, and haptic feedback control commands based on the music events. The communication gateway module sends each control command to the audio system, ambient lighting controller, in-vehicle display system, and haptic feedback actuator. If the vehicle status changes or a safety function is triggered, the system exits the full music mode or downgrades to a low-interference feedback mode.

[0061] In some embodiments, multiple striking points may be arranged within the vehicle cabin, as can be seen from [reference needed]. Figure 3This includes: the entire front row, the passenger-side dashboard, the left front door panel, the left rear door panel, the right front door panel, the right rear door panel, the brake pedal, the floor mats (front row), and the front and rear seats. It includes designated areas of focus, general focus, basic structural areas, and non-structural areas (optional). Specific tapping points can be added or removed based on the vehicle's structure and material characteristics.

[0062] See again Figure 2 Steering wheel tap detection can reuse HOD / vibration / capacitive sensors. Door panel / dashboard tap detection can identify taps on different interior panels using piezoelectric / IMU sensors or structured sound microphones. Armrest / seat / carpet detection can identify tapping motions and foot rhythm input using pressure / film / vibration sensors. Brake pedal read-only detection can be achieved using pressure / displacement / pedal cover sensors. Visual perception can also be achieved using a DMS camera, specifically identifying tap locations or dynamic gestures.

[0063] In some embodiments, the communication link design of the vehicle cockpit system is as follows: Figure 4 As shown, this application adopts a layered communication architecture. The parameters acquired by the sensing and acquisition layer include capacitance, piezoelectricity, vibration, pressure, displacement, and those from cameras or microphones. Specifically, the sensing and acquisition layer is responsible for converting HOD, capacitance, piezoelectricity, vibration, pressure array, pedal displacement, visual data from the DMS camera, or acoustic signals from the in-vehicle microphone into electrical signals. The sensing and acquisition layer communicates with the area control layer via local ADC, SPI, and I2C. The area control layer is responsible for near-end filtering, threshold judgment, and event preprocessing; the cockpit orchestration layer completes time synchronization, event recognition, safety gating, and music / lighting orchestration through the cockpit domain controller or interactive orchestration controller. The execution control layer drives audio, ambient lighting, vehicle display, and haptic feedback according to instructions. The steering wheel controller can report steering wheel tapping events to the cockpit domain controller via LIN, CAN, or CAN FD. Door panel controllers, seat controllers, and pedal sensing units can transmit tap data to the gateway or cockpit domain controller via LIN, CAN, CAN FD, or local I / O. The cockpit domain controller can communicate with the vehicle system via Ethernet, SOME / IP, or internal IPC, with the audio system via A2B, I2S, TDM, or Ethernet AVB, and with the ambient lighting controller via LIN, CAN, PWM, or SPI.

[0064] In some embodiments, the music mapping and multimodal linkage process can be as follows: Figure 5As shown, after security gating allows, standardized knock events (valid knock events, Knock_Event) are converted into musical events and distributed in parallel to the music mapping engine and the scene and score engine. The music mapping engine can determine parameters such as note, timbre, dynamics, and tempo. The scene and score engine supports three working modes: free play, score-guided, and multi-person ensemble. Specifically, the timbre is determined based on the knocking area, the volume is determined based on the knocking force, the tempo is determined based on the knocking rhythm, and the sound image position is determined based on the location of the knocking component in the cockpit space. For example, the steering wheel can be mapped to a melody or piano note, the door panel can be mapped to a drum kit or percussion, the armrest can be mapped to a bass, the brake pedal can be mapped to a bass drum or a strong beat, and the carpet can be mapped to a low-frequency ambient sound or ripple effect.

[0065] The output data from both engines is aggregated to the linkage orchestrator. The orchestrator processes the data based on a unified tempo, timestamp, and motion curve, while simultaneously issuing control commands to: the audio output unit (outputting notes, samples, and sound image signals); the ambient lighting (controlling color, brightness, and flow); the vehicle infotainment system (displaying animations, sheet music, and highlight effects); and haptic feedback on the seats, steering wheel, and armrests. The system also integrates a recording and learning module to save user preferences and rhythm templates. The same tapping event can simultaneously drive multi-channel outputs of sound, light, screen, and touch. For example, if a user taps the left door panel, the left door panel light flows towards the center console area, and sound is generated from the left speaker array; if a user taps the brake pedal, the low-level floor light diffuses towards the front cabin, and the low-frequency speaker outputs a bass drum; if multiple users tap from different seats, the system can create an independent music track for each seat and mix the outputs.

[0066] Specifically, the above process can be referred to Table 1.

[0067] Table 1

[0068] In some embodiments, the vehicle cockpit control method of this application can be applied to the vehicle cockpit system described above, or to the controller of the vehicle cockpit system described above, and this application does not impose any specific limitations on it.

[0069] like Figure 6 As shown, the vehicle cockpit control method of this application includes: S601. When the target function is activated, in response to user interaction input, control the vehicle's audio system to output audio signals.

[0070] The target function represents a functional mode that generates corresponding audio signals based on user interactive input. When this function is active, the interactive input is recognized and responded to by the vehicle's controller, thereby outputting an audio signal. In this application, the target function can be simply understood as the function of outputting custom music through interaction. In one possible implementation, the target function can be activated upon detecting that the user has triggered at least one of the following: a preset button press, a gesture, and / or a voice command.

[0071] In this application, the target function can operate as a scene mode within the vehicle infotainment system. The target function may include at least one of the following modes: performance mode, ensemble mode, and sheet music guidance mode. Performance mode is used to instruct the mapping of cockpit components and / or interactive areas to different musical notes to simulate instrumental performance. Ensemble mode is used to instruct the mapping of different cockpit components to different instrument types to simulate multi-instrument ensemble performance. Sheet music guidance mode is used to guide the user to perform different interactive inputs to play sheet music.

[0072] Interactive input refers to the operation commands initiated by the user to the in-vehicle cockpit interaction system. Interactive input can be a tactile action acting on physical components of the cockpit or a non-contact sensing action. In this application, interactive input can include at least one of the following: tapping, patting, light touching, heavy tapping, continuous rhythmic actions, pedaling actions, voice, gestures, and swiping. For example, tactile operations such as the driver tapping the steering wheel interaction area, the passenger tapping the door panel interior, or pressing the brake pedal, or non-contact operations such as in-vehicle voice commands or camera-recognized air gestures, can all serve as interactive input.

[0073] One possible implementation is to control the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin.

[0074] Cockpit components refer to the parts inside the vehicle's cabin that users can tap or touch, such as the steering wheel, interior panels, seats, dashboard, armrests, carpets, brake pedals, etc.

[0075] A touch action represents a contact-based operation in which a user applies force to a cockpit component, causing the component to vibrate. Touch actions fall under the category of contact-based interactive inputs mentioned above, and differ from non-contact actions such as gestures and voice commands. In this application, touch actions include at least one of the following: tapping, slapping, light tapping, heavy tapping, continuous rhythmic movements, stomping, etc.

[0076] Audio devices refer to devices that can emit sound inside and outside a vehicle. In this application, they can include all audio output devices such as speakers and headrest speakers inside and outside the vehicle. For example, an audio device can be a speaker or horn inside the vehicle, or an external speaker unit outside the vehicle, etc.

[0077] In one possible implementation, the controller determines whether the target function is activated. If not, it does not respond to touch actions to avoid accidental triggering. If it is activated, it collects tap signals using sensors such as vibration sensors, acceleration sensors, and pressure sensors located on cabin components to identify the user's touch action. The tap signals are then filtered to eliminate other vibration interference. Upon receiving a valid tap signal, the controller generates a corresponding audio control command and sends it to the vehicle audio system, driving the system to output the corresponding audio signal.

[0078] In one implementation, after receiving a valid tap signal, the controller generates a corresponding audio control command. This can be achieved by generating different audio control commands based on the tapping location, number of taps, and tapping force, thereby outputting different audio content. The system pre-establishes a mapping relationship between tapping features and audio control commands: the tapping location is determined based on the location of the tapping signal, such as different cabin components like the steering wheel, door, and armrest. The number of consecutive valid taps is counted based on the tapping signal to obtain the tapping count, and the tapping force level is determined based on the vibration amplitude and duration. The controller combines the multi-dimensional features of location, number of taps, and force to match the preset audio control commands.

[0079] For example, when the vehicle has the target function activated, if the driver lightly taps the steering wheel, the vibration sensor detects the touch action, and the controller controls the vehicle audio system to play a short feedback sound effect or output music.

[0080] Therefore, when the target function is activated, by converting the user's interactive input into audio signals for the audio device, audio signals are generated based on the user's real-time operation. Compared with the existing technology that can only play audio, this application breaks away from the limitation of one-way intelligent audio playback through real-time response between interactive input and audio output. It supports drivers and passengers to actively create audio interactive content, enriches the form of cabin entertainment, enhances the fun of cabin interaction and the level of vehicle intelligence, thereby improving the experience of drivers and passengers.

[0081] The target functions of this application will be described in detail below, using user interaction input as a touch action.

[0082] In some embodiments, the vehicle includes a plurality of cabin components; each cabin component includes a plurality of interaction areas. One cabin component corresponds to a musical instrument; and the interaction area of ​​one cabin component corresponds to a voice part of the musical instrument.

[0083] The cockpit components and the final output audio signal in this application implementation include two layers of mapping relationship: First layer: There is a one-to-one correspondence between the cockpit components and musical instruments, thereby ensuring that the same component corresponds to the timbre of one type of instrument and that the same component does not simultaneously function as two different instruments. Second layer: The interactive areas inside the cockpit components correspond one-to-one with the independent voices under that instrument, thereby enabling the triggering of different areas to output different voice effects.

[0084] For example, the steering wheel corresponds to a piano. The steering wheel is divided into multiple interactive areas, each corresponding to one of the piano's middle register: Do, Re, Mi, Fa, Sol, La, and Si. Striking different areas outputs different piano notes.

[0085] (1) Performance mode In some embodiments, when the cockpit component is a steering wheel, S601 can be implemented as follows: determining the target area on the steering wheel where the touch action is applied; the target area is one or more of a plurality of interactive areas on the steering wheel; generating a corresponding audio signal based on the target note or target chord corresponding to the target area, and controlling the audio device to output the audio signal; wherein, one interactive area corresponds to one note or one chord.

[0086] The interactive area on the steering wheel represents the touch areas divided on the steering wheel, and each area can perform a touch action.

[0087] The target area refers to the area actually affected by the user's touch action. It can be a single area or multiple areas that are tapped simultaneously.

[0088] As one possible implementation, the target note or target chord corresponding to the target region is determined based on a preset region-note mapping relationship or a preset region-chord mapping relationship, and the corresponding audio signal is generated based on the target note or target chord.

[0089] The region-note mapping relationship represents the pre-configured and stored correspondence rules between regions and notes. For example, an interactive region corresponds to a note (Do, Re, Mi, etc.).

[0090] The region-chord mapping relationship represents the pre-configured and stored correspondence rules between regions and chords. For example, one interactive region corresponds to one chord, or multiple regions can be combined to form a chord when triggered simultaneously.

[0091] In one possible implementation, the steering wheel is pre-divided into several independent interactive areas. When a user's touch action on the steering wheel is detected, a tap signal is collected. Based on the sensor position corresponding to the tap signal, the target area corresponding to this touch action is determined. The preset mapping rules include: (1) area-note mapping, where one interactive area corresponds to a single note; tapping the corresponding area matches the target note; (2) area-chord mapping, where one interactive area directly corresponds to a set of chords. The target note or target chord is obtained by matching based on the preset mapping rules and the target area. Based on the target note or target chord, a corresponding control command is generated and sent to the audio device to drive the audio device to play the target note / target chord.

[0092] If different interactive areas are tapped in sequence, different notes can be output in sequence according to the order of the actions to achieve the melody of the song. If multiple areas are tapped at the same time, the sounds can be superimposed to form a compound harmony.

[0093] It should be understood that by transforming the steering wheel into a performance input device, different notes and chords can be produced by striking different areas, thus enabling drivers and passengers to play musical instruments based on the steering wheel without the need for additional independent instruments. This enriches the forms of in-vehicle entertainment and interaction, and enhances the driving experience.

[0094] For example, such as Figure 7 As shown, in the zoned performance mode, the steering wheel is divided into 8 interactive zones (Z1-Z8), each zone is bound to a note (Do, Re, Mi, Fa, etc.). When the driver taps the upper left zone of the steering wheel while driving, the system identifies the target zone, matches the note Sol, and controls the car audio system to play the Sol note. Tapping different interactive zones sequentially outputs notes to form a melody. Different musical role zones can be mapped to piano scales, drum beats, chords, or sampled sound effects in different timbre modes. Continuously swiping through different musical role zones corresponds to glissando or rapid tapping. To improve stability, when the zone confidence level is below a threshold, the system can downgrade the specific zone note mapping to a general beat event.

[0095] It should be noted that in the embodiments of this application, binding different interactive areas with different notes and chords does not constitute a limitation of this application, and binding areas with scales, drumbeats, sampled sound effects, etc. is also within the protection scope of this application.

[0096] (2) Ensemble mode In some embodiments, the touch action includes multiple actions, wherein some of the multiple touch actions are triggered simultaneously or sequentially. S601 described above can be implemented by determining the cockpit components corresponding to the multiple touch actions; generating an audio signal based on the instrument type corresponding to each touch action; and controlling the audio device to output the audio signal.

[0097] As one possible implementation, the cockpit components corresponding to multiple touch actions are determined; based on the preset mapping relationship between cockpit components and musical instruments, the type of musical instrument corresponding to each touch action is determined; based on the type of musical instrument, an audio signal is generated, and the audio device is controlled to output the audio signal.

[0098] The mapping relationship between cabin components and musical instruments represents the pre-configured matching rules stored between cabin components and musical instruments. For example, the steering wheel in the driver's seat is mapped to the main melody instrument such as the high register of a piano, the right door panel trim panel (with an embedded piezoelectric sensor) in the passenger seat is mapped to a snare drum, the rear door panel is mapped to a tom-tom, the door armrest is mapped to a bass, the rear carpet area is mapped to a kick drum, and the brake pedal is mapped to the bass drum or strong beat controller of the whole band.

[0099] As one possible implementation, the user generates multiple touch actions within a preset time (e.g., 5 seconds), where the multiple touch actions can be triggered simultaneously or sequentially according to time order.

[0100] Sensors collect tapping signals from various cockpit components and send them to the controller. The controller analyzes the signals to determine which cockpit component each tapping action occurred on. It then queries a pre-defined mapping between cockpit components and musical instruments: each type of cockpit component corresponds to one type of instrument, thus determining the instrument type matched to each tapping action. If multiple tapping actions are triggered simultaneously, multiple instrument timbres are superimposed to create an ensemble audio. If multiple tapping actions are triggered sequentially, audio signals are generated and played according to the order of the taps, based on the corresponding instrument timbres.

[0101] For example, suppose the steering wheel corresponds to a piano, the passenger-side door panel corresponds to a snare drum, and the rear door panels are mapped to a tom-tom. When the driver strikes the steering wheel (piano notes), the passenger and rear passengers simultaneously strike the door panels (drum beats). The system recognizes the simultaneous striking actions and superimposes the piano, snare drum, and tom-tom timbres into the output audio signal. In this way, the system identifies the cabin components corresponding to each striking action, synthesizes multiple instrument audios in parallel, and enables collaborative performance by multiple people inside the vehicle.

[0102] It should be understood that different cabin components correspond to different musical instruments, and passengers can switch the timbre by striking different positions, which can simulate a band performance, thereby enriching the forms of in-vehicle cabin entertainment and interaction and enhancing the driving experience.

[0103] (3) Sheet music guidance mode In some embodiments, one interactive area corresponds to one note; the above S601 can be implemented as follows: determining the score to be played, the score to be played includes: multiple notes arranged in sequence; based on the score to be played, controlling the vehicle's display device and / or lighting device to output guidance instructions, the guidance instructions are used to prompt the user to perform a touch action in the corresponding interactive area; based on the touch action performed by the user, generating an audio signal and controlling the audio device to output.

[0104] See again Figure 7 In the score progression mode, the system pre-stores scores consisting of multiple music events E1, E2, E3...E5. Each music event can include pitch, timbre, duration, dynamics, voice type, lighting parameters, and tactile parameters. Each valid tap by the user outputs the music event currently pointed to by the score cursor and advances the cursor to the next music event. The system can also perform beat quantization and misbeat compensation based on the time interval between adjacent taps.

[0105] The sheet music to be played represents a series of note data stored internally by the vehicle's infotainment system, arranged in chronological order of performance. When the sheet music guidance mode is activated, users can select the sheet music they wish to play through manual selection, voice interaction, or other means, thus enabling them to perform the piece.

[0106] Guidance instructions are prompts output by the vehicle's infotainment system to the user, informing them which interactive area to tap. Guidance instructions can be displayed through ambient lighting strips, light spots on the screen, or other means.

[0107] Display devices include: in-vehicle displays, instrument panel displays, HUDs, and other devices capable of outputting graphic and textual prompts.

[0108] The lighting system includes ambient lights, indicator lights, and other in-vehicle light source components that can output light, which can illuminate or illuminate target interactive areas.

[0109] As one possible implementation, after the target function is activated, the user selects the sheet music to be played. The controller determines the stored data corresponding to the sheet music based on the user's operation. After parsing the stored data, the controller obtains a sequence of notes arranged chronologically. Based on the correspondence between notes and interactive areas, it finds the target interactive area corresponding to the note (the target interactive area differs for different instruments). Text and graphic guidance pops up on the display device, and / or the lighting device illuminates the corresponding interactive area, thus outputting guidance instructions. The user taps the target interactive area according to the guidance. The sensor continuously collects the tapping signal, identifies the user's touch action, matches it with the corresponding note, generates an audio signal, and controls the sound device to emit sound. Then, based on the above process, it proceeds to the next note of the sheet music until the entire piece is played.

[0110] It should be noted that the guiding instructions may include an interactive area corresponding to a single note in the musical score, or an interactive area corresponding to multiple notes; this application does not impose any specific restrictions on this.

[0111] For example, when a user selects to play the sheet music "Twinkle Twinkle Little Star," the system parses the notes into a sequential sequence and uses the ambient lights embedded in the steering wheel to illuminate the corresponding interactive areas of the notes in the order of the performance, guiding the driver to tap them. After the driver taps the illuminated area, the audio system plays the corresponding note, completing the entire piece of music.

[0112] Therefore, by determining the score to be played and then outputting guiding instructions, the user is guided to trigger touch actions in sequence according to the score, thus realizing the ability to play along with the score. This expands the forms of in-vehicle music interaction and enhances the entertainment of driving. In particular, the embodiments of this application can support novices to complete the playing of music by following the guidance, enriching the interactive gameplay in the cabin and improving the passenger experience.

[0113] In some embodiments, S601 can be implemented as follows: in response to user interaction input, control the vehicle's audio device to output an audio signal, and control at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device.

[0114] As one possible implementation, when the target function is active, the sensor identifies valid interactive input from the user on the cockpit components, the controller generates audio control commands, and drives the audio device to output corresponding audio signals. Simultaneously, linkage control commands are output to drive at least one of the lighting device, display device, and haptic feedback device to operate.

[0115] For example, when a passenger taps on the door panel, the audio system outputs a drumbeat, and at the same time, the haptic feedback device built into the seat vibrates briefly, achieving audio and tactile linkage. Alternatively, when a user taps on the armrest box to trigger a chord, the ambient lighting strip around the interior lights up instantly according to the tapping location. The color of the light spot changes with the pitch, mapping to different colors, and can flow along the edges towards the center to form a visual melody line, achieving video and visual linkage.

[0116] It should be understood that by controlling lighting devices, display devices, haptic feedback devices and other devices in conjunction with audio devices, a multi-dimensional sensory system can be constructed. During the audio signal output process, composite signals such as visual, tactile and auditory signals are output simultaneously, which enhances the immersive feeling of the performance interaction and improves the passenger experience.

[0117] The following describes the linkage process of this application.

[0118] (1) The lighting device includes at least one of the following: ambient light, indicator light, interior door lighting module, etc. The above linkage process can be specifically implemented as follows: determine the light emission parameters of the lighting device according to the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input; the light emission parameters include at least: brightness, color, flashing frequency or lighting sequence; based on the light emission parameters, control the lighting device and the audio device to link together.

[0119] As one possible implementation, audio parameters can include notes, volume, timbre, and tempo, while the spatial location information corresponding to the interactive input is used to locate the position where the interactive action occurs. Based on a pre-stored mapping relationship between audio parameters, interactive positions, and lighting parameters, when the audio is played, the corresponding visual effects are synchronously driven to output lights, achieving sound following light movement and matching the interactive position with the light position.

[0120] In one implementation, when the interactive input is a touch signal, the light emission parameters of the lighting device are determined based on the audio parameters represented by the audio signal and / or the interactive area corresponding to the touch action; the light emission parameters include at least: brightness, color, flashing frequency or lighting sequence; based on the light emission parameters, the lighting device and the sound device are controlled to work together.

[0121] For example, when a driver taps an interactive area on the steering wheel, triggering a high-pitched piano note audio, the indicator light corresponding to that interactive area on the steering wheel instantly brightens and switches to a cool color tone, with the duration of the light matching the duration of the note playback. If the tapping force is greater and the audio volume increases, the light brightness increases simultaneously. When an occupant taps an interactive area on the door panel, triggering a drum beat sound effect, the door panel light strip flashes in accordance with the drum beat; with continuous and rapid tapping, the flashing frequency of the light increases synchronously.

[0122] (2) The display devices include: in-vehicle central control screen, instrument panel screen, head-up display, rear-seat display screen, and seat back display screen. The above linkage process can be specifically implemented as follows: based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input, the display parameters of the display device are controlled. The display parameters include at least: color, brightness, display position or display content; based on the display parameters, the display device and the audio device are linked.

[0123] As one possible approach, based on the spatial location information of the interactive input, the cabin area where the interactive input occurs is determined, and the specific display location of the display device is determined, such as the in-vehicle central control screen, instrument screen, or rear display screen for image output. Based on the audio parameters corresponding to the audio signal, the display parameters such as display content, color, brightness, and dynamic effects are adjusted and displayed on the display interface to remind passengers of the currently triggered note information, the corresponding instrument type, the playing progress, and the striking force.

[0124] In one implementation, when the interactive input is a touch signal, the display parameters of the display device are controlled according to the audio parameters represented by the audio signal and / or the interactive area corresponding to the touch action. The display parameters include at least: color, brightness, display position or display content; based on the display parameters, the display device and the audio device are controlled to work together.

[0125] For example, when a user presses the brake pedal to trigger the bass part audio, the central control screen displays the brake pedal and bass instrument icon, and simultaneously displays the current bass note. When the user taps the interactive area of ​​the left front door trim panel to play the snare drum beat, the rear display screen displays the striking animation. In the sheet music guidance mode, the central control screen displays the sheet music, marks the playing progress according to the touch action, and switches the prompt color to remind the user when a playing mistake is made.

[0126] (3) The haptic feedback device includes at least one of the following: suspension, seat, steering wheel, pedal. The above linkage process can be specifically implemented as follows: based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input, the haptic parameters of the haptic feedback device are controlled, and the haptic parameters include at least: vibration intensity, vibration frequency or vibration timing; based on the haptic parameters, the haptic feedback device is controlled to link with the audio device.

[0127] One possible approach is to extract audio frequency, volume, and tempo information from the audio parameters and match the corresponding tactile parameters according to a preset mapping relationship. Specifically, the lower the audio frequency, the lower the vibration frequency; for heavy beats in music, the vibration intensity is increased accordingly. Based on spatial location information, the position of the tactile feedback device that needs to trigger vibration feedback is determined, such as the steering wheel, brake pedal, or corresponding seat. Based on the position and tactile parameters of the tactile feedback device, the vibration of the tactile feedback device is controlled.

[0128] In one implementation, when the interactive input is a touch signal, the tactile parameters of the tactile feedback device are controlled according to the audio parameters represented by the audio signal and / or the interactive area corresponding to the touch action. The tactile parameters include at least: vibration intensity, vibration frequency, or vibration timing. Based on the tactile parameters, the tactile feedback device is controlled to link with the audio device.

[0129] Specifically, when the driver taps the interactive area of ​​the steering wheel to play a melody note, the corresponding vibrating component on the steering wheel activates simultaneously, with the vibration duration matching the audio playback duration. When a forceful tap produces a loud sound effect, the vibration intensity increases accordingly. In a multi-person ensemble setting, touch actions at different locations drive corresponding vibrating components such as the seat to provide tactile feedback.

[0130] It should be understood that by synchronously linking the audio system with the lighting, display, and haptic feedback devices, a multi-dimensional collaborative feedback system encompassing auditory, visual, and tactile senses is constructed. In music interaction scenarios, drivers and passengers perceive the corresponding music signals, lighting changes, visual feedback, and vibration feedback for each interaction input from multiple dimensions, thereby enhancing the immersive experience of the interaction.

[0131] The following sections describe the process of the audio device outputting audio signals and its interaction with other devices in several scenarios.

[0132] Example 1 (Performance Scenario) When the vehicle is parked (in P gear, at zero speed, with the electronic parking brake engaged and the driver relaxed), the existing hardware resources of the steering wheel can be utilized to provide the driver with an immersive solo melody creation experience. After the user selects the steering wheel melody mode via the central control screen or activates the target function via voice command, the system first verifies the vehicle status gating module to confirm that the safe entertainment conditions are met, including being in P gear, at zero speed, and without any emergency braking signals.

[0133] The system divides the steering wheel rim into multiple virtual semitone zones along its circumference, or the steering wheel spokes into multiple chord trigger zones, such as the C major tonic chord or the G major dominant chord. The system reuses the steering wheel's internal capacitance sensor to monitor the rate of change in capacitance values ​​in each zone in real time. It combines this with high-frequency vibration signals collected by the steering wheel's built-in inertial measurement unit and corrects the signal using a camera. The recognition algorithm identifies continuous capacitance changes as a grip, and short, rapid capacitance changes accompanied by specific frequency vibration peaks as valid tapping events (i.e., valid tapping signals).

[0134] Based on the physical area where the strike occurs, such as the upper left or lower right, it maps to the corresponding note or chord, and calculates the intensity of the note based on the peak amplitude of the vibration signal, achieving the effect of a soft strike for a soft sound and a hard strike for a loud sound. It loads a piano or synthesizer sound library by default and allows users to switch to guitar or string sounds. In terms of multimodal linkage feedback, the ambient light strip around the interior illuminates instantly with the strike location, and the color of the light spot changes with the pitch, mapping to different colors, and can flow from the edge to the center to form a visual melody line.

[0135] Example 2 (Ensemble Scene) In scenarios such as vehicle parking and charging, camping, or multiple people resting, the controller can automatically identify the number and location of occupants inside the vehicle, mapping different cabin components to different musical instruments to create a multi-person band environment. For example, the driver's steering wheel is mapped to a main melody instrument such as the high register of a piano, the piezoelectric sensor embedded in the right door panel of the passenger side is mapped to a snare drum, the rear door panels are mapped to a tom-tom, the door armrests are mapped to a bass, the rear carpet area is mapped to a kick drum, and the brake pedal is mapped to the bass drum or a strong beat controller for the entire band.

[0136] The cockpit domain controller uses Ethernet or the vehicle's high-bandwidth bus to assign microsecond-level unified timestamps to tapping events reported by various nodes, including door panels, armrests, carpets, and pedals. This eliminates delays caused by different signal transmission paths, ensuring strict rhythmic synchronization during multi-person ensemble playing. The system dynamically adjusts the volume balance of each track based on the input density of each instrument, preventing one instrument from drowning out other parts. Users can select rock, jazz, or electronic styles on the in-vehicle system, and the system automatically loads the corresponding sound library and adjusts reverb and delay parameters.

[0137] Example 3 (Music score guidance scenario) For users lacking music theory knowledge, visual guidance and algorithmic assistance lower the barrier to entry for playing, enabling even beginners to play complete melodies. After the user selects a target song, such as "Twinkle Twinkle Little Star," on the central control screen, the system loads a simplified MIDI score. The ambient light strip or a light spot on the screen moves towards the target tapping area 0.5 seconds in advance according to the rhythm of the song, creating a visual effect similar to falling notes in a rhythm game.

[0138] If the user taps within an error window (e.g., ±50 milliseconds) of the light spot reaching the designated area, the system plays the standard note, the light changes color to confirm, and the interaction score increases. If the user's tap is slightly earlier or later (e.g., within ±100 milliseconds), the system activates a beat quantization algorithm to automatically align the note timestamp to the nearest standard beat grid, ensuring a smooth rhythm in the output music and eliminating artificial rhythmic instability. If the user taps the wrong area, the system automatically replaces the incorrect note with a correct neighboring note based on the probability model of the context melody, or simply plays a soft transitional sound effect to avoid harsh mispronunciations that could ruin the experience.

[0139] When the system detects that the user has tapped correctly in a series of consecutive times, it triggers a combo effect, the ambient light gradually warms up, and drum beats and bass accompaniment are automatically added to the background music to enhance the user's sense of accomplishment.

[0140] In some embodiments, S601 can be implemented as follows: determining the trigger time of the touch action, and for multiple touch actions at the same time, synthesizing the sub-audio signals corresponding to each touch action to generate an audio signal and outputting it.

[0141] The trigger time refers to the moment when each touch action occurs, or the moment it is detected by the sensor. The system can set a preset time window, for example, 100ms. Primary actions within the preset time window are judged as occurring at the same moment.

[0142] The sub-audio signal is the audio signal generated by each touch action, which can be a note, a chord, a sound effect, etc.

[0143] As one possible implementation, the controller determines whether a tap action has occurred and the trigger time of each tap action based on the tap signals collected by the sensor. If multiple tap actions fall within the same time window (at the same moment), the controller determines the sub-audio signal corresponding to each tap action, mixes and synthesizes the multiple sub-audio signals to generate a composite signal, i.e., an audio signal. If multiple tap actions fall within different time windows (not at the same moment), the controller outputs each sub-audio signal sequentially.

[0144] For example, the driver simultaneously taps two different areas on the steering wheel. The two taps are triggered at the same time. Area A matches the note Do, and area B matches the note Mi. The system generates two sub-audio signals, Do and Mi, respectively, mixes them to form a composite signal for output. That is, the output audio signal includes both Do and Mi.

[0145] Therefore, by mixing and synthesizing multiple sub-audio signals corresponding to the same moment, the synchronous sound production of multiple keys on a real musical instrument can be simulated, thereby enriching the forms of musical expression.

[0146] In some embodiments, the cockpit components corresponding to the touch action include multiple components; the above S601 can be implemented as follows: based on the touch action corresponding to each cockpit component, determine the sub-audio signal corresponding to each cockpit component, synthesize the sub-audio signals to obtain an audio signal and output it.

[0147] One possible approach is to identify touch actions corresponding to multiple cockpit components, generate a sub-audio signal for each struck cockpit component independently, mix all sub-audio signals into a composite audio signal, and control the audio output.

[0148] Therefore, by synthesizing the corresponding sub-audio signals of different cabin components, it is possible to simulate a multi-instrument collaborative performance scenario, thereby improving the expressive form of in-vehicle music interaction.

[0149] In some embodiments, S601 above can be implemented as determining the type of musical instrument corresponding to each cockpit component, and determining the sub-audio signal corresponding to the touch action based on the type of musical instrument, so that multiple musical instruments output audio signals based on the same beat.

[0150] The beat represents the timing reference, which is reflected in the starting time of multiple instruments, rhythm speed, etc. The same beat means that all instruments share a unified timing clock reference, and the playing speed is kept consistent, avoiding the rhythm of each instrument being scattered and out of sync.

[0151] One possible approach is to determine the type of instrument bound to each struck cockpit component, generate a corresponding sub-audio signal based on the timbre characteristics of that instrument type, and assign a unified time reference to each sub-audio signal so that each sub-audio signal is aligned with the same beat on the time axis.

[0152] For example, the steering wheel corresponds to a piano, and the left front door trim panel corresponds to a drum kit. The driver taps the steering wheel, and the front passenger simultaneously taps the left front door trim panel. The system generates piano sub-audio and drum sub-audio separately, aligns the two audio streams with a unified rhythm, mixes them, and plays the piano melody and drum beats simultaneously to form an audio signal with a unified rhythm.

[0153] Understandably, since different instruments produce sound differently, unifying the rhythm can ensure that each part is precisely aligned on the time axis, avoiding rhythmic deviations caused by differences in the onset delay and decay time of different instruments, thus ensuring the rhythmic consistency and coordination of each instrument during ensemble playing.

[0154] In some embodiments, the cockpit components corresponding to the touch action include multiple components; the above-mentioned S601 can be implemented by adjusting the volume parameters of the sub-audio signals corresponding to each cockpit component based on the number of touch actions corresponding to each cockpit component within a preset time period, so as to achieve volume balance.

[0155] The preset duration refers to a pre-defined statistical time window (e.g., within 3 seconds) used to count the frequency of taps.

[0156] The number of striking actions represents the total number of times a single cockpit component is struck within a time window, characterizing the level of performance involvement of that instrument's vocal part.

[0157] Volume balance: When playing multi-part ensemble, avoid one instrument being too loud and drowning out other instruments, and maintain auditory balance among all parts.

[0158] One possible approach is to determine the number of times each cockpit component is tapped within a preset time period. A higher number of taps indicates a more active current sound, requiring volume control to avoid masking other sounds. Based on the mapping rule between the tap frequency of each cockpit component and a preset volume, the volume gain of each sub-audio signal is adjusted. Then, the multiple sub-audio signals are mixed and synthesized to achieve a balanced overall ensemble volume.

[0159] For example, the steering wheel (piano) is struck 5 times in 2 seconds, while the left door panel (drum kit) is struck only 2 times. The piano part is struck more frequently, and the volume parameters of the piano sub-audio are reduced to ensure that the drum beats are not completely drowned out by the piano melody.

[0160] Therefore, by statistically analyzing the touch frequency of each interactive cockpit component, the volume of each audio path is dynamically adjusted to achieve volume balance in multi-instrument ensemble scenarios, thereby improving the clarity of sound and the performance experience in in-vehicle ensemble mode.

[0161] In some embodiments, the cockpit components corresponding to the touch action include multiple components; the above-mentioned S601 can be implemented as follows: based on the target music style, determine the corresponding timbre parameters, reverberation parameters and / or delay parameters; generate an audio signal and output it according to the timbre parameters, reverberation parameters and / or delay parameters.

[0162] The target music style refers to the genre selected by the user or automatically matched by the system, such as rock, jazz, traditional Chinese music, electronic music, etc.

[0163] As one possible approach, the target music style is determined, and based on the preset correspondence between style and audio parameters, the corresponding timbre parameters, reverberation parameters, delay parameters, etc., are determined. When generating the sub-audio signals corresponding to each cockpit component, audio rendering is performed based on the timbre parameters, reverberation parameters, and / or delay parameters to obtain the final audio signal and output it.

[0164] Therefore, by matching timbre parameters, reverberation parameters, and / or delay parameters based on the target music style, the same cabin components can output sound effects adapted to different music styles, thereby enriching the forms of musical expression and enhancing the in-vehicle performance interaction experience for drivers and passengers.

[0165] In some embodiments, S601 can be implemented as follows: when the difference between the output time of the guidance instruction and the trigger time of the touch action is less than a first preset duration threshold, the sound device is controlled to output a note corresponding to the guidance instruction, and the light device is controlled to output light feedback.

[0166] The output time of the guidance command indicates the point in time when the light / display device prompts the user to tap the target area. The trigger time of the touch action indicates the point in time when the sensor detects the tapping action.

[0167] The first preset duration threshold represents the maximum allowable time deviation for whether the user's trigger action is a timely performance; for example, it can be 50ms.

[0168] As one possible implementation, the output time of the guidance instruction and the trigger time of the touch action are determined, and the time difference between the output time of the guidance instruction and the trigger time of the touch action is calculated. If the time difference is less than the first preset duration threshold, the user's action is determined to be a timely tap, the standard note corresponding to the guidance instruction is played, and the lighting device is triggered to output positive light feedback.

[0169] For example, the system illuminates the Mi area light on the steering wheel (guidance command). If the user taps the area within 10ms and the difference is less than a first preset duration threshold, the audio system plays the Mi note, and the corresponding area light on the steering wheel flashes white light as feedback.

[0170] In some embodiments, the above-mentioned S601 can be implemented as follows: when the difference between the trigger time of the touch action and the output time of the guidance instruction is greater than or equal to a first preset duration threshold and less than a second preset duration threshold, the output time of the audio signal corresponding to the touch action is corrected to a preset time, where the preset time is the closest musical score beat point to the trigger time.

[0171] The second preset duration threshold represents the critical value at which a slight delay is determined and calibration can still be performed; for example, it can be 100ms.

[0172] As one possible implementation, if the time difference is greater than or equal to the first preset duration threshold and less than the second preset duration threshold, it indicates that the user's touch action has a delay, but has not completely missed the beat. In this case, the note is not played immediately, but the audio is output at the standard beat point of the score closest to the current tap.

[0173] For example, if the user's tap time is slightly ahead or behind, for example within plus or minus one hundred milliseconds, the system enables the beat quantization algorithm to automatically align the note timestamps to the nearest standard beat grid, ensuring smooth rhythm in the output music and eliminating human-induced rhythm instability.

[0174] It should be understood that by setting a first preset duration threshold that is less than a second preset duration threshold, this application can perform real-time correct feedback or correction processing according to the time deviation between the user's touch action and the guidance instruction, thereby ensuring that the performance is guided while solving the problem of rhythm deviation caused by slow user operation and improving the smoothness of playing the score.

[0175] In some embodiments, S601 can be implemented as follows: when the actual tapping area corresponding to the touch action does not match the target tapping area corresponding to the guidance instruction, the incorrect note corresponding to the actual tapping area is replaced with the target note corresponding to the target tapping area, and / or a transition sound effect is output to replace the incorrect note.

[0176] One possible approach is to compare the actual striking area with the target striking area corresponding to the guiding instruction. If they do not match, the striking position is determined to be incorrect. In this case, the incorrect note can be masked and forcibly replaced with the standard note corresponding to the target area. Alternatively, the incorrect note can be discarded, and a preset transition sound effect can be played to fill the gap.

[0177] For example, the prompt suggests tapping the Sol area. If the user accidentally taps the Do area, the target Sol note can be output directly, or a smooth glissando transition sound effect can be played.

[0178] Therefore, by adjusting for tapping errors, the continuity of the musical performance can be avoided by avoiding accidental triggering of touch actions, thereby optimizing the smoothness of playing the score and improving the user experience.

[0179] In some embodiments, S601 can be implemented as follows: in response to a user's touch action on a cockpit component of the vehicle, controlling the audio device in the vehicle to output an audio signal, including: if the number of consecutive correct taps by the user is greater than a preset number, performing at least one of the following operations: outputting a continuous tap sound effect; controlling the lighting device in the vehicle to adjust the color temperature of the lights; controlling the audio device to add drum beats and / or bass parts to the audio signal.

[0180] As one possible implementation, the controller counts the number of correct taps. When the number exceeds a preset threshold (e.g., 5 consecutive correct taps), it can output a combo sound effect, change parameters such as color temperature and rhythm, and automatically add drum beats and bass accompaniment to the original audio quotation marks to enhance the user's interactive experience.

[0181] For example, the preset threshold is 5 consecutive correct taps; when the user taps 5 times in a row, a short burst of sound effects is triggered, the steering wheel ambient light switches from cool white to warm yellow, and a bass accompaniment is automatically added.

[0182] Therefore, by recognizing continuous and correct tapping behavior and providing multi-dimensional interactive feedback, the fun of playing music together can be enhanced, thereby improving the user experience.

[0183] In some embodiments, S601 can be implemented as follows: responding to a user's touch action sequence on a vehicle's cabin component; determining a note sequence corresponding to the touch action sequence based on the touch action sequence; and controlling the audio device in the vehicle to output an audio signal based on the note sequence.

[0184] Therefore, the sequence of actions generated by a user continuously tapping different cockpit components can be converted into a sequence of melodic notes and output, realizing a melody creation scenario.

[0185] In some embodiments, S601 described above can be implemented as follows: determining the audio parameters of the audio to be generated based on the interactive parameters of the interactive input; and arranging and generating an audio signal based on the audio parameters of the audio to be generated and outputting it.

[0186] Audio parameters are used to characterize at least one of the following: note, timbre, volume, beat, sound image position, and sound effect of the audio to be generated; touch parameters include at least one of the following: touch position, force, duration, and speed.

[0187] Among them, musical notes: such as Do / Re / Mi, etc.; timbre: the timbre of instruments such as piano, drums, guitar, bass, etc.; volume: the loudness of the sound; tempo: the rhythm and speed of the music signal; sound image location: the location of the sound in the car's sound field (front, back, left, right); sound effects: reverb, delay, etc.

[0188] As one possible implementation, the audio parameters to be generated can be determined based on the interactive parameters of the interactive input. This can be achieved by parsing the interactive features and matching the corresponding note, timbre, volume, tempo, sound image, and sound effect parameters based on the mapping relationship between the interactive parameters and the audio parameters.

[0189] In one implementation, when the interactive input is a touch action, the interactive parameters are touch parameters; the touch parameters include at least one of the following: the position, force, duration, and speed of the touch action; wherein, the timbre and / or note is determined based on the interactive area corresponding to the touch action; the volume is determined based on the force of the touch action; the beat is determined based on the frequency of the touch action; and the sound image position is determined based on the position of the cockpit component corresponding to the touch action in the vehicle cockpit.

[0190] The tapping parameters represent data describing the characteristics of tapping behavior, obtained from the analysis of tapping signals collected by sensors. These parameters include: Location: the cockpit component or interaction area where the tapping action occurred; Force: the amplitude of the tapping signal; Duration: the duration of the tapping signal; and Speed: the rate of change of the tapping signal, used to identify different rhythms such as rapid taps and slow taps.

[0191] In other words, the above can be implemented as follows: based on the touch parameters of the touch action, the audio parameters to be generated are determined. Specifically, the touch signal corresponding to the touch action is collected by a sensor, and the touch parameters (position, force, duration, and tapping frequency) corresponding to this touch action are extracted. Audio parameters are obtained by matching the touch parameters one by one: the touch position (corresponding to the interaction area) determines the note and timbre; the touch force determines the volume; the touch frequency determines the tempo; and the spatial position of the cabin component determines the audio image position. Based on all audio parameters, the audio signal is synthesized in real time and output by the vehicle audio system.

[0192] For example, tapping the interactive area on the left side of the steering wheel matches a musical note or piano timbre based on the position; a larger tapping force corresponds to a higher volume, with the steering wheel located in the center of the front of the cabin, and the sound image set to the center of the cabin, outputting a medium-length piano note with each tap.

[0193] Therefore, by analyzing the interaction parameters to determine the audio parameters, the logic of real instrument playing can be restored, allowing users to control audio parameters such as the volume, timbre, rhythm, and sound image position of notes through characteristic interactive input, thereby enhancing the immersive experience of the performance.

[0194] In some embodiments, S601 described above can be implemented as follows: acquiring an interactive signal generated by the interactive input through an on-board sensor; extracting features from the interactive signal to determine interactive features; using the interactive features to characterize the signal quality of the interactive signal; determining the interactive signal as a valid interactive signal if the interactive feature characterization meets a preset quality condition; and controlling the audio device in the vehicle to output an audio signal based on the valid interactive signal.

[0195] In one implementation method, the method of acquiring the interaction signal includes at least one of the following: acquiring it through a sensor corresponding to the steering wheel; acquiring it through a sensor corresponding to the inner panel and / or dashboard; acquiring it through a sensor corresponding to the armrest and / or seat and / or carpet; acquiring it through a sensor corresponding to the brake pedal; acquiring it through a camera; or acquiring it through a microphone.

[0196] It should be understood that the interaction signal is a signal directly acquired by the vehicle-mounted sensor without processing and may include interference signals. This application characterizes the signal attributes of the interaction signal by extracting interaction features and comparing them with preset conditions. This allows for the selection of valid interaction signals from the original signal, providing a reliable basis for the subsequent audio signal generation.

[0197] As one possible implementation, when the interactive input is a touch action, the above process can be implemented as follows: the vehicle-mounted sensor acquires the tapping signal corresponding to the touch action, the tapping signal is feature extracted to determine the tapping feature; the tapping feature is used to characterize the vibration response generated by the touch action on the cabin component, and if the tapping feature characterizes the vibration generated by the touch action, the tapping signal is determined to be a valid tapping signal, and based on the valid tapping signal, the audio device in the vehicle is controlled to output an audio signal.

[0198] In one possible implementation, the method of acquiring the tapping signal corresponding to the touch action through an on-board sensor includes: acquiring the initial signal corresponding to the touch action through the on-board sensor; determining the force deviation signal based on the limb's falling height and speed; correcting the initial signal based on the force deviation signal to obtain the corrected initial signal; and preprocessing the corrected initial signal to obtain the tapping signal. The preprocessing includes at least one of the following: filtering, noise reduction, and normalization.

[0199] The following example will illustrate this point.

[0200] In some embodiments, the tap event recognition method is as follows: Figure 8As shown, the standardized processing flow of the impact signal first performs multi-source signal acquisition to obtain multiple raw sensor signals C(t) / A(t) / P(t) / M(t). The raw signals are sent to the preprocessing module for filtering, noise reduction, and time synchronization. Subsequently, candidate impact detection is performed to extract signal peak value, rising edge, and duration information. Then, feature extraction is performed to obtain features such as peak amplitude, rising edge slope, vibration decay time, spectral center, short-time energy, arrival time difference of multiple sensors, pressure duration, and continuous impact interval. The system completes identification based on valid impact criteria, including short-time peak value exceeding a threshold, rising edge slope satisfying the impact model, vibration decay conforming to component material characteristics, and arrival time difference consistent with neighboring nodes. Based on the extracted features, multi-source fusion judgment is performed to identify the component and region corresponding to the impact and calculate the confidence level. Finally, a standardized impact event containing region, force, rhythm, and timestamp information is generated. Standardized tap events are encapsulated into a Knock_Event structure, whose fields include Event_ID, Node_ID, Zone_ID, Part_Type, Intensity, Rhythm_Interval, Timestamp, Confidence, and Vehicle_State. The events output by this process are decoupled from specific sensors, which facilitates unified processing of subsequent music mapping and multimodal linkage.

[0201] In some embodiments, the brake pedal corresponds to a first transmission link and a second transmission link. The first transmission link and the second transmission link are isolated from each other. The first transmission link is used to transmit braking signals. The second transmission link is used to transmit interaction signals.

[0202] Mutual isolation means electrical or logical isolation between two links. Signal abnormalities or algorithm failures in the second transmission link will not affect the first transmission link, ensuring that braking-related signals are not interfered with by interactive functions.

[0203] As mentioned above, the brake pedal can serve as a cockpit component for interactive input, and the acquisition of primary movements is based on brake pedal sensors. These sensors include pedal displacement sensors, force sensors, and acceleration sensors, which can collect pedal travel, pedal vibration, and pedal force signals to provide relevant data for vehicle braking.

[0204] In this application, by adding a transmission link while keeping the first transmission link unchanged, a new parallel signal channel is added, which is isolated from the first transmission link; this channel is specifically used to transmit the interaction signals required for the tapping interaction.

[0205] Therefore, by setting up a first transmission link and a second transmission link that are mutually isolated, it is possible to isolate the entertainment interaction function from the vehicle's braking safety control system while reusing the brake pedal sensor to recognize pedal taps, thus avoiding brake failure caused by abnormal interaction function.

[0206] In one implementation, the input signal of the first transmission link includes the acquisition signal of the brake pedal sensor, and the interaction signal transmitted by the second transmission link is a copy of the acquisition signal in the first transmission link.

[0207] It should be understood that the second transmission link is a copy of the signal from the first transmission link, and cannot be modified or intruded upon. This ensures that the interaction signal and the original braking acquisition signal are from the same source, avoiding motion recognition errors caused by signals from different sources.

[0208] The following example illustrates this embodiment.

[0209] Brake pedal entertainment input isolated from safety link, such as Figure 9 As shown, this application employs a read-only acquisition method for the brake pedal node. Pedal pressure, pedal displacement, or independent pedal cover sensor signals can enter the cockpit entertainment link (first transmission link) through an isolation gateway or independent acquisition channel, but the entertainment link must not write control commands to the braking execution link (first transmission link). The braking control link is still independently completed by the brake ECU, hydraulic brake, or brake-by-wire system, with braking safety always taking precedence over music interaction. When the brake pedal signal simultaneously meets both real braking and music interaction needs, the system prioritizes the real braking demand, and music interaction is masked or interrupted. This design allows the brake pedal to be used as a rhythm input in parking entertainment mode, such as bass drum, strong beat, beat progression, or music pause input, while avoiding interference with the original braking system.

[0210] Specifically, the brake pedal assembly features a dual-path signal acquisition architecture. Path A is the safety control path, retaining the traditional brake pedal displacement or pressure sensor. The signal is directly sent to the electronic brake control unit for calculating braking torque and controlling the hydraulic or brake-by-wire system. This path has the highest priority. Path B is the entertainment input path, with the signal sent separately to the cockpit domain controller solely for recognizing rhythmic pedal movements.

[0211] The two signal software protocol stacks operate independently. The entertainment link has only read access to the braking system and absolutely no write access. The cockpit domain controller cannot send any control commands to the electronic brake control unit (the entertainment link only reads and collects pedal signals; under no circumstances should it write control commands to the braking execution link). In parking entertainment mode, the signal of path B is only activated and mapped to a music event when the vehicle is in P gear, the vehicle speed is zero, and the electronic parking brake is locked. Quickly pressing the brake pedal is mapped to a bass drum to simulate the sound of the kick drum in jazz, producing a deep and powerful sound. In score-guided mode, pressing the pedal represents a strong beat in the music, and the system uses this to advance the rhythm. Lightly pressing the pedal without triggering the brake switch travel can trigger special bass effects or reverb effects.

[0212] In some embodiments, the steering wheel corresponds to a third transmission link and a fourth transmission link. The third and fourth transmission links are isolated from each other. The signal transmitted by the third transmission link is used for driver grip detection; the fourth transmission link is used to acquire and transmit interaction signals.

[0213] The meaning of mutual isolation can be found in the description in the above embodiments and will not be repeated here.

[0214] The steering wheel can serve as a cockpit component for interactive input, and touch gestures are collected using steering wheel sensors. These sensors can be HOD sensors, which can collect data on the driver's grip, steering wheel vibration, and force signals, providing relevant data for the vehicle driver monitoring system.

[0215] In this application, by adding a transmission link, the third transmission link retains its original function and continuously transmits the driver's grip detection signal, and a new parallel signal channel, namely the fourth transmission link, is added, which is isolated from the third transmission link and is specifically used to transmit the interaction signal required for the tapping interaction.

[0216] Therefore, by setting up mutually isolated third and fourth transmission links, the original vehicle steering wheel sensor can be reused to realize the recognition of tapping actions in the steering wheel area, while isolating the cockpit entertainment interaction function from the driver monitoring function to avoid interference of the interaction signal with the driver's grip detection.

[0217] In some embodiments, the input signal of the third transmission link includes the acquisition signal of the sensor corresponding to the steering wheel, and the interaction signal transmitted by the fourth transmission link is a copy of the acquisition signal in the third transmission link.

[0218] It should be understood that the fourth transmission link only reads and copies the same source signal as the third communication link, and cannot modify or interfere with the original sensing signal in the third transmission link. This ensures that the interactive recognition signal and the original acquisition signal of the grip detection are from the same source, avoiding action recognition deviations caused by different signal sources.

[0219] like Figure 10 As shown, the first detection signal S1 output by the HOD controller 110 enters the safety link 200 (equivalent to the third transmission link mentioned above) for safety-related functions, including ADAS, vehicle safety functions, hands-off alarm, and takeover warning. The cockpit music interaction function does not directly modify the first detection signal, nor does it write back to the HOD controller 110. Instead, it obtains the second detection signal S2 through the music bypass link 300 (equivalent to the fourth transmission link mentioned above) for tap recognition and performance control. The second detection signal S2 can be a copy of the first detection signal or a non-safety event tag output by the HOD controller.

[0220] The isolation constraints include: 1) The music link only reads or receives copies of the HOD signal and does not participate in the HOD security loop; 2) The music link must not modify the HOD controller's judgment thresholds, fault diagnosis strategies, and safety output logic; 3) When the HOD status is abnormal, the driver assistance takeover reminder is triggered, or the driving risk status is triggered, the music link will be automatically turned off or downgraded.

[0221] In one implementation, the second detection signal S2 is sent to the cockpit domain controller via a non-security data frame. Its data content may include area touch state, transient touch state, touch duration, touch intensity, touch confidence level, or touch release state. The cockpit domain controller uses this data only to execute entertainment interaction logic, without altering the thresholds, fault diagnosis strategies, and output logic in the HOD safety judgment link 200.

[0222] In some embodiments, S601 can be implemented as follows: when the vehicle meets preset safety conditions, in response to user interaction input, control the vehicle's audio device to output an audio signal.

[0223] The preset safety conditions include: the vehicle is in P gear; the vehicle speed is zero; the vehicle is in parking brake mode; the vehicle's braking control function is not activated; the vehicle's steering control function is not in emergency takeover mode; and the vehicle's active safety functions are not activated.

[0224] In some embodiments, the method further includes: when the vehicle does not meet preset safety conditions and the source of the interactive input is the driver's position, controlling the second transmission link corresponding to the brake pedal to not transmit the initial signal, and / or controlling the fourth transmission link corresponding to the steering wheel to not transmit the initial signal, and / or controlling the target function to be downgraded or turned off.

[0225] Therefore, when the driver does not meet the safety requirements, blocking the trigger path of the target function from the driver's position and controlling the target function to be downgraded or turned off can ensure that the driver is not disturbed by non-driving tasks during driving and ensure driving safety.

[0226] Considering that the steering wheel and brake pedal are safety-related components, this application includes a vehicle status safety gating module. Steering wheel and pedal tapping events are only allowed to enter the music interaction chain when the vehicle is in a safe entertainment scenario (meeting preset safety conditions). Safe entertainment scenarios may include being in P gear, at zero speed, with the parking brake engaged, brake control not dynamically activated, steering control not in emergency takeover mode, and vehicle safety control functions such as AEB, ABS, and ESC (active safety) not activated.

[0227] When the vehicle is in D or R gear, the vehicle speed is greater than zero, the driver takeover reminder is activated, active safety functions are triggered, emergency braking or steering assist is activated, the system prohibits the steering wheel and pedals from being used as music input, or exits the full music mode, and only allows the vehicle screen to display prompts, low-brightness ambient lighting rhythm or passive music playback.

[0228] Specifically, such as Figure 11 As shown, the safety gating process first receives a tap event, then reads vehicle status information including vehicle speed, gear position, parking brake, brake activation, and driving task priority to determine if the current operating condition meets the safety conditions for complete music playback (preset safety conditions). If the conditions are not met, i.e., the vehicle is in motion, not in park, or the brake is activated, the steering wheel and pedal music playback functions are disabled, retaining only low-interference feedback or directly exiting the music playback link. Among these, braking, steering, AEB, ABS, and ESC safety functions have the highest priority. If the conditions are met, i.e., the vehicle is in parking entertainment, camping, or charging waiting conditions, then multiple component tap signals are allowed to enter the music link, and the system outputs Allow_Flag and Mode_ID signals. This process follows the gating principle: entertainment input must not change the vehicle's safety execution link, and safety control takes precedence in the event of a functional conflict.

[0229] In some embodiments, the above method further includes: when the vehicle does not meet the preset safety conditions, the source of the interactive input is a non-driver's position, and the touch action is applied to the steering wheel, when the vehicle speed is less than or equal to the preset vehicle speed, the control target function operates in a restricted mode; or when the vehicle speed is greater than the preset vehicle speed, the control target function is downgraded or turned off.

[0230] By differentiating vehicle speeds, the behavior of non-driver passengers touching the steering wheel is controlled in a tiered manner. At low speeds, non-driver passengers are allowed to use the target function in a restricted mode (such as only allowing mono output), while at high speeds, the function is further restricted or completely turned off. This ensures driving safety while preserving the target function as much as possible to improve the passenger experience.

[0231] In some embodiments, where the cockpit components include only the steering wheel, such as Figure 12 As shown, the sensor mentioned above is the steering wheel HOD sensor 100.

[0232] The steering wheel HOD sensor 100 collects the contact signal of the driver's hand relative to the steering wheel, and the HOD controller 110 generates a first detection signal based on the contact signal. The first detection signal enters the HOD safety judgment link 200 (equivalent to the third transmission link mentioned above) for driver grip status judgment, hands-off warning, or assisted driving takeover warning. At the same time, the signal bypass module 300 (equivalent to the music bypass link mentioned above, or the fourth transmission link) bypasses and copies or reads the first detection signal to generate a second detection signal. The second detection signal is input to the vehicle status gating module 400, the tapping event recognition module 500, and the performance control module 600.

[0233] It should be noted that the first detection signal S1 enters the HOD security judgment link; the second detection signal S2 is only used for music interaction, and the music link does not write back, nor does it modify the HOD security, diagnostic strategy, and security output logic.

[0234] The performance control module 600 executes steering wheel-based zone performance or score progression based on the permission results output by the vehicle status gating module 400 and the valid tapping events output by the tapping event recognition module 500. The feedback output module 700 controls the vehicle audio system 710, cabin ambient lighting 720, and haptic feedback device 730 to output synchronously based on the music events generated by the performance control module 600. The feedback output module 700 can convert each music event into audio output parameters, lighting output parameters, and haptic output parameters. Audio output parameters are sent to the vehicle audio system 710, lighting output parameters are sent to the cabin ambient lighting 720, and haptic output parameters are sent to the haptic feedback device 730. Lighting output parameters may include starting position, propagation direction, brightness, color, duration, diffusion speed, and attenuation curve. The haptic feedback intensity can be determined based on the music event intensity or tapping intensity level.

[0235] like Figure 13As shown, the method of this embodiment includes the following steps: S101 Acquire a first HOD detection signal; S102 Generate a second detection signal by bypass copying; S103 Acquire the vehicle operating status; S104 Determine music interaction permissions; S105 Identify valid tapping events based on the second detection signal; S106 Determine the performance mode; S107 Perform steering wheel zone performance or music score cursor advancement; S108 Generate music events; S109 Output audio, light, and / or haptic feedback; S110 Record and save in-vehicle music segments.

[0236] In some embodiments, the interactive input is a touch action; the interactive signal is a tapping signal; when the cockpit component is a steering wheel, a valid tapping signal is determined by: filtering tapping signals acquired by sensors corresponding to the steering wheel based on a first condition to obtain candidate tapping signals; the first condition includes: the amplitude of the tapping signal is greater than a preset amplitude threshold, the rate of change of the amplitude of the tapping signal is greater than a preset rate threshold, and the duration of the tapping signal is less than a preset tapping time; filtering the candidate tapping signals based on a second condition to obtain a valid tapping signal; the second condition includes: the target grip parameter is less than or equal to a preset grip parameter threshold; the target grip parameter includes: grip duration and / or grip contact area; The time difference between the triggering time of the tapping signal and the triggering time of the steering wheel button signal is less than or equal to the first time difference; the time difference between the triggering time of the tapping signal and the occurrence time of the vehicle vibration signal is less than or equal to the second time difference.

[0237] The tapping event recognition module can perform baseline calibration, filtering, and transient change detection on the second detection signal. Since differences in steering wheel material, ambient temperature, humidity, steering wheel heating, hand sweat, and driver physical condition can cause capacitance baseline drift, the system can update the baseline using a sliding window, low-pass filtering, or temperature compensation. When a signal in a certain area experiences a rapid rise and rapid release within a preset time window, and the duration is within a preset tapping time range, the system identifies it as a candidate tapping event.

[0238] Candidate tapping events need further filtering for false touches. If the driver is continuously holding the area when the event occurs, and the contact area or duration exceeds the corresponding threshold, the event is judged as a change in grip rather than a tap. If the event is highly correlated with steering wheel button operation or vehicle vibration, it can be judged as button interference or road vibration interference. Candidate events that meet the amplitude threshold, slope threshold, duration threshold, and confidence threshold are determined as valid tapping events.

[0239] like Figure 14As shown, the vehicle status gating module can divide steering wheel music interaction permissions into three levels: fully open, restricted open, and closed or downgraded. The gating principle is: as the driving risk level increases, the steering wheel music interaction permissions decrease step by step; when a safety risk event is triggered, the music interaction is immediately terminated, blocked, or downgraded. G1 Fully Open: When the vehicle is in P gear parking, charging, camping, cinema, or display vehicle mode, or when the vehicle speed is 0 and there is no alarm, the system opens the full steering wheel music function. G2 Restricted Open: When the vehicle is in a low-speed driving scenario, the system can only allow click prompts, music progress, or passenger area interaction, or reduce the volume and lighting effects. G3 Closed or Downgraded: When the vehicle is in a medium-to-high speed driving scenario, when Advanced Driver Assistance Systems (ADAS) are activated, when the driver's attention is abnormal, when there is a HOD alarm, when there is a takeover warning, when there is emergency braking, or when there is a collision risk, the system closes or downgrades the steering wheel music interaction function.

[0240] It should be noted that the aforementioned steering wheel zoned performance mode is primarily designed for steering wheel hardware with multi-area touch recognition capabilities. In practical engineering applications, this solution is also fully compatible with steering wheels that have a single-zone structure or only support overall contact detection. In this case, the system can simply enable the score progression mode, mapping the entire steering wheel as a single tap input area. Users can trigger valid tap events by tapping any valid area of ​​the steering wheel. The system then drives the score cursor sequentially and outputs preset music events, thus achieving low-threshold, highly stable melody performance interaction without the need for complex zoned hardware.

[0241] Furthermore, this technical solution is not only applicable to conventional fixed steering wheels but also fully compatible with foldable steering wheel architectures. Throughout the unfolding, folding, and retracting process of the steering wheel, the data communication link between the HOD sensor array and the cockpit domain controller remains independent. The mechanical displacement and deformation of the folding mechanism do not interfere with the acquisition, bypass replication, and subsequent signal processing logic of the first detection signal. The system can monitor the steering wheel's shape and state in real time in conjunction with the vehicle state gating module and adaptively optimize baseline calibration and filtering parameters during the folding action to ensure the accuracy of impact event recognition. The steering wheel folding process does not substantially affect the core implementation steps of the method described in this patent, such as signal bypass multiplexing, spectral progression, and state gating, ensuring the universality and robustness of this interaction solution under different steering wheel configurations.

[0242] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the vehicle cockpit control device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] This application embodiment can, based on the above method, exemplarily divide the control device of a vehicle cockpit into functional modules. For example, the control device of the vehicle cockpit may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0244] Reference Figure 15 The vehicle cockpit control device 1500 includes: a processing unit 1501.

[0245] The processing unit 1501 is used to control the vehicle's audio system to output audio signals in response to user interaction input when the target function is activated.

[0246] Optionally, the interactive input includes: a touch action; the processing unit 1501 is specifically used to control the vehicle's audio device to output an audio signal in response to the user's touch action on a cockpit component of the vehicle.

[0247] Optionally, the vehicle includes multiple cabin components; each cabin component includes multiple interaction areas; wherein, one cabin component corresponds to one musical instrument; and the interaction area of ​​one cabin component corresponds to a part of one musical instrument.

[0248] Optionally, when the cockpit component is a steering wheel, the processing unit 1501 is specifically used to determine the target area on the steering wheel where the touch action is performed; the target area is one or more of multiple interactive areas on the steering wheel; based on the target note or target chord corresponding to the target area, a corresponding audio signal is generated, and the audio device is controlled to output the audio signal; wherein, one interactive area corresponds to one note or one chord.

[0249] Optionally, the touch action includes multiple actions, wherein some of the multiple touch actions are triggered simultaneously or sequentially; the processing unit 1501 is specifically used to determine the cockpit components corresponding to the multiple touch actions; based on the type of musical instrument corresponding to each touch action, an audio signal is generated, and the audio device is controlled to output the audio signal.

[0250] Optionally, one interactive area corresponds to one note; the processing unit 1501 is specifically used to determine the score to be played, which includes: multiple notes arranged in sequence; based on the score to be played, control the vehicle's display device and / or lighting device to output guidance instructions, which are used to prompt the user to perform a touch action in the corresponding interactive area; based on the touch action performed by the user, generate an audio signal and control the audio device to output.

[0251] Optionally, the processing unit 1501 is specifically used to respond to the user's interactive input, control the vehicle's audio device to output audio signals, and control at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device.

[0252] Optionally, the processing unit 1501 is specifically used to determine the light emission parameters of the lighting device based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input; the light emission parameters include at least: brightness, color, flashing frequency or lighting sequence; and based on the light emission parameters, control the lighting device and the sound device to work together.

[0253] Optionally, the processing unit 1501 is specifically used to control the display parameters of the display device based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input. The display parameters include at least: color, brightness, display position or display content; and based on the display parameters, control the display device to link with the audio device.

[0254] Optionally, the processing unit 1501 is specifically used to control the tactile parameters of the tactile feedback device according to the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input. The tactile parameters include at least: vibration intensity, vibration frequency or vibration timing; and control the tactile feedback device to link with the audio device based on the tactile parameters.

[0255] Optionally, the processing unit 1501 is specifically used to determine the audio parameters of the audio to be generated based on the interactive parameters of the interactive input; the audio parameters are used to characterize at least one of the notes, timbre, volume, beat, sound image position, and sound effects of the audio to be generated; and to arrange and generate an audio signal based on the audio parameters of the audio to be generated and output it.

[0256] Optionally, when the interactive input is a touch action, the interactive parameters are touch parameters; the touch parameters include at least one of the following: the position, force, duration, and speed of the touch action; the timbre and / or note is determined based on the interactive area corresponding to the touch action; the volume is determined based on the force of the touch action; the beat is determined based on the frequency of the touch action; the sound image position is determined based on the position of the cockpit component corresponding to the touch action in the vehicle cockpit; the sound image position is used to characterize the position of the audio to be generated in the stereo field.

[0257] Optionally, the touch action includes multiple actions; the processing unit 1501 is specifically used to determine the trigger time of the touch action; for multiple touch actions at the same time, the sub-audio signals corresponding to each touch action are synthesized to generate an audio signal and output it.

[0258] Optionally, the cockpit components corresponding to the touch action include multiple components; the processing unit 1501 is specifically used to determine the sub-audio signal corresponding to each cockpit component based on the touch action corresponding to each cockpit component; synthesize the sub-audio signals to obtain an audio signal and output it.

[0259] Optionally, the processing unit 1501 is specifically used to determine the type of musical instrument corresponding to each cockpit component; based on the type of musical instrument, it determines the sub-audio signal corresponding to the touch action, so that multiple musical instruments output audio signals based on the same beat.

[0260] Optionally, the cockpit components corresponding to the touch action include multiple components; the processing unit 1501 is specifically used to adjust the volume parameters of the sub-audio signals corresponding to each cockpit component based on the number of touch actions corresponding to each cockpit component within a preset time period, so as to achieve volume balance.

[0261] Optionally, the cockpit components corresponding to the touch action include multiple components; the processing unit 1501 is specifically used to determine the corresponding timbre parameters, reverberation parameters and / or delay parameters based on the target music style; and to generate and output an audio signal based on the timbre parameters, reverberation parameters and / or delay parameters.

[0262] Optionally, the processing unit 1501 is specifically used to control the sound device to output the note corresponding to the guidance instruction and control the lighting device to output light feedback when the difference between the output time of the guidance instruction and the trigger time of the touch action is less than a first preset duration threshold.

[0263] Optionally, the processing unit 1501 is further configured to correct the output time of the audio signal corresponding to the touch action to a preset time when the difference between the trigger time of the touch action and the output time of the guidance instruction is greater than or equal to a first preset duration threshold and less than a second preset duration threshold. The preset time is the closest musical score beat point to the trigger time.

[0264] Optionally, the processing unit 1501 is specifically used to replace the erroneous note corresponding to the actual interaction area with the target note corresponding to the target interaction area when the actual interaction area corresponding to the touch action does not match the target interaction area corresponding to the guidance instruction, and / or output a transition sound effect to replace the erroneous note.

[0265] Optionally, the processing unit 1501 is specifically used to perform at least one of the following operations when the number of consecutive correct taps by the user exceeds a preset number: outputting a continuous tap sound effect; controlling the vehicle's lighting device to adjust the color temperature of the lights; and controlling the audio device to add drum beats and / or bass parts to the audio signal.

[0266] Optionally, the processing unit 1501 is specifically configured to respond to a user's touch action sequence on a vehicle's cabin component; determine a note sequence corresponding to the touch action sequence based on the touch action sequence; and control the vehicle's audio device to output an audio signal based on the note sequence.

[0267] Optionally, the processing unit 1501 is specifically used to acquire the interactive signal generated by the interactive input through the vehicle-mounted sensor; extract features from the interactive signal to determine the interactive features; the interactive features are used to characterize the signal quality of the interactive signal; if the interactive feature characterization meets the preset quality conditions, the interactive signal is determined to be a valid interactive signal; based on the valid interactive signal, the audio device in the vehicle is controlled to output an audio signal.

[0268] Optionally, the interactive signals may be acquired through at least one of the following methods: via a sensor corresponding to the steering wheel; via a sensor corresponding to the inner panel and / or dashboard; via a sensor corresponding to the armrest and / or seat and / or carpet; via a sensor corresponding to the brake pedal; via a camera; or via a microphone.

[0269] Optionally, the interactive input is a touch action; the interactive signal is a tapping signal; when the cockpit component is a steering wheel, the processing unit 1501 is specifically used to filter the tapping signals obtained by the sensors corresponding to the steering wheel based on a first condition to obtain candidate tapping signals; the first condition includes: the amplitude of the tapping signal is greater than a preset amplitude threshold, the rate of change of the amplitude of the tapping signal is greater than a preset rate threshold, and the duration of the tapping signal is less than a preset tapping time; the candidate tapping signals are filtered based on a second condition to obtain valid tapping signals; the second condition includes: the target grip parameter is less than or equal to a preset grip parameter threshold; the target grip parameter includes: grip duration and / or grip contact area; the time difference between the triggering time of the tapping signal and the triggering time of the steering wheel button signal is less than or equal to a first time difference; the time difference between the triggering time of the tapping signal and the occurrence time of the vehicle vibration signal is less than or equal to a second time difference.

[0270] Optionally, the brake pedal corresponds to a first transmission link and a second transmission link; the first transmission link and the second transmission link are isolated from each other; the first transmission link is used to transmit braking signals; and the second transmission link is used to transmit interactive signals.

[0271] Optionally, the input signal of the first transmission link includes the sensor acquisition signal of the brake pedal, and the interaction signal transmitted by the second transmission link is a copy of the acquisition signal in the first transmission link.

[0272] Optionally, the steering wheel corresponds to a third transmission link and a fourth transmission link; the third transmission link and the fourth transmission link are isolated from each other; the signal transmitted by the third transmission link is used for driver grip detection; the fourth transmission link is used to acquire and transmit interactive signals.

[0273] Optionally, the input signal of the third transmission link includes the acquisition signal of the sensor corresponding to the steering wheel, and the interaction signal transmitted by the fourth transmission link is a copy of the acquisition signal in the third transmission link.

[0274] Optionally, the processing unit 1501 is specifically used to control the vehicle's audio device to output an audio signal in response to the user's interactive input when the vehicle meets preset safety conditions; the preset safety conditions include: the vehicle is in P gear; the vehicle speed is zero; the vehicle is in parking brake state; the vehicle's braking control function is not activated; the vehicle's steering control function is not in emergency takeover state; and the vehicle's active safety functions are not activated.

[0275] Optionally, the processing unit 1501 is further configured to, when the vehicle does not meet the preset safety conditions and the source of the interactive input is the driver's position, control the second transmission link corresponding to the brake pedal not to transmit the initial signal, and / or control the fourth transmission link corresponding to the steering wheel not to transmit the initial signal, and / or control the target function to be downgraded or turned off.

[0276] Optionally, the processing unit 1501 is further configured to, when the vehicle does not meet the preset safety conditions, the source of the interactive input is a non-driver's position, and the touch action is applied to the steering wheel, operate the target control function in a restricted mode when the vehicle speed is less than or equal to the preset vehicle speed; or, downgrade or shut down the target control function when the vehicle speed is greater than the preset vehicle speed.

[0277] Optionally, the cabin components include at least one of the following: steering wheel, interior panel, seat, dashboard, armrest, carpet, and brake pedal.

[0278] Optionally, interactive input may include at least one of the following: tapping, slapping, light touching, heavy tapping, continuous rhythmic movements, stomping movements, voice, gestures, or swipes.

[0279] Optionally, the target function is a function mode that generates corresponding audio signals based on user interactive input.

[0280] Optionally, the target function includes at least one of the following modes: performance mode; performance mode is used to instruct the mapping of cockpit components and / or interactive areas to different notes to simulate instrument playing; ensemble mode; ensemble mode is used to instruct the mapping of different cockpit components to different instrument types to simulate multi-instrument ensemble playing; sheet music guidance mode; sheet music guidance mode is used to instruct and guide the user to perform different interactive inputs to play sheet music.

[0281] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 16 As shown, the electronic device 1600 includes, but is not limited to, a processor 1601 and a memory 1602.

[0282] The memory 1602 described above is used to store the executable instructions of the processor 1601. It is understood that the processor 1601 is configured to execute instructions to implement the vehicle cockpit control method in the above embodiments.

[0283] It should be noted that those skilled in the art will understand that Figure 16 The device structure shown does not constitute a limitation on the device; the device may include, but is not limited to, other types of devices. Figure 16 This may indicate more or fewer components, or a combination of certain components, or a different arrangement of components.

[0284] Processor 1601 is the control center of electronic device 1600. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in memory 1602, and by calling data stored in memory 1602, thereby providing overall monitoring of the device. Processor 1601 may include one or more processing units. Optionally, processor 1601 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 1601.

[0285] The memory 1602 can be used to store software programs and various data. The memory 1602 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, processing unit, etc.), etc. Furthermore, the memory 1602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage component, flash memory component, or other volatile solid-state storage component.

[0286] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1602 including instructions, which can be executed by a processor 1601 of an electronic device 1600 to implement the methods in the above embodiments.

[0287] In actual implementation, Figure 15 The processing unit 1501 in the middle can be made by Figure 16 The processor 1601 calls the computer program stored in the memory 1602 to implement the process. The specific execution process can be found in the method section of the previous embodiment, and will not be repeated here.

[0288] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0289] In an exemplary embodiment, this application also provides a vehicle including the aforementioned device and computer-readable storage medium.

[0290] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by the processor 1601 of the device to perform the methods described above.

[0291] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of the device, they implement the various processes of the above method embodiments and achieve the same technical effect as the above method. To avoid repetition, they will not be described again here.

[0292] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0293] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0294] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the classified units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0295] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0296] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or a complete or partial classification of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0297] In the description of the embodiments of this application, specific features, structures, materials or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0298] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for controlling a vehicle cockpit, characterized in that, The method includes: When the target function is activated, the system controls the vehicle's audio system to output audio signals in response to user input.

2. The method according to claim 1, characterized in that, The interactive input includes: a touch action; the response to the user's interactive input, controlling the vehicle's audio device to output an audio signal includes: In response to a user's touch action on a component of the vehicle's cabin, the system controls the vehicle's audio system to output an audio signal.

3. The method according to claim 2, characterized in that, The vehicle includes multiple cockpit components; each cockpit component includes multiple interaction areas. In this configuration, one cockpit component corresponds to one musical instrument; and the interaction area of ​​one cockpit component corresponds to one voice part of one musical instrument.

4. The method according to claim 3, characterized in that, When the cockpit component is the steering wheel, the audio device controlling the vehicle outputs an audio signal, including: The target area on the steering wheel is determined by the touch action; the target area is one or more of the plurality of interactive areas on the steering wheel; Based on the target note or target chord corresponding to the target area, a corresponding audio signal is generated, and the audio device is controlled to output the audio signal; Each of the interactive areas corresponds to a note or a chord.

5. The method according to claim 3, characterized in that, The touch action includes multiple actions, some of which are triggered simultaneously or sequentially. The audio device controlling the vehicle outputs an audio signal, including: Identify the cockpit components corresponding to each of the aforementioned touch actions; Based on the instrument type corresponding to each of the aforementioned touch actions, the audio signal is generated, and the audio device is controlled to output the audio signal.

6. The method according to claim 3, characterized in that, One interactive area corresponds to one musical note; The method of controlling the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin includes: Determine the musical score to be played, which includes: multiple notes arranged in sequence; Based on the sheet music to be played, the vehicle's display device and / or lighting device are controlled to output guiding instructions, which are used to prompt the user to perform a touch action in the corresponding interactive area. Based on the user's touch action, an audio signal is generated and the audio device is controlled to output the signal.

7. The method according to any one of claims 1-6, characterized in that, The step of controlling the vehicle's audio system to output an audio signal in response to user interaction input includes: In response to user interaction input, the system controls the vehicle's audio device to output the audio signal, and controls at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device.

8. The method according to claim 7, characterized in that, The audio device controlling the vehicle outputs the audio signal, and controls at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device, including: The light emission parameters of the lighting device are determined based on the audio parameters represented by the audio signal and / or the spatial location information corresponding to the interactive input; the light emission parameters include at least: brightness, color, flashing frequency or lighting sequence; Based on the light emission parameters, the lighting device and the sound device are controlled to work together.

9. The method according to claim 7, characterized in that, The audio device controlling the vehicle outputs the audio signal, and controls at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device, including: Based on the audio parameters represented by the audio signal and / or the spatial location information corresponding to the interactive input, the display parameters of the display device are controlled, and the display parameters include at least: color, brightness, display position or display content; Based on the display parameters, the display device and the audio device are controlled to work together.

10. The method according to claim 7, characterized in that, The audio device controlling the vehicle outputs the audio signal, and controls at least one of the lighting device, display device, and haptic feedback device to be linked with the audio device, including: Based on the audio parameters represented by the audio signal and / or the spatial position information corresponding to the interactive input, the tactile parameters of the tactile feedback device are controlled, wherein the tactile parameters include at least: vibration intensity, vibration frequency, or vibration timing. Based on the tactile parameters, the tactile feedback device and the audio device are controlled to work together.

11. The method according to any one of claims 1-10, characterized in that, The step of controlling the vehicle's audio system to output an audio signal in response to user interaction input includes: Based on the interactive parameters of the interactive input, the audio parameters of the audio to be generated are determined; the audio parameters are used to characterize at least one of the following: note, timbre, volume, beat, sound image position, and sound effect of the audio to be generated. Based on the audio parameters of the audio to be generated, the audio signal is arranged, generated, and output.

12. The method according to claim 11, characterized in that, When the interactive input is the touch action, the interactive parameter is the touch parameter; the touch parameter includes at least one of the following: the position, force, duration, and speed of the touch action; The timbre and / or the note are determined based on the interactive area corresponding to the touch action; The volume is determined based on the force of the touch action; The beat is determined based on the frequency of the striking action; The sound image position is determined based on the position of the cockpit component corresponding to the touch action in the vehicle cockpit; the sound image position is used to characterize the position of the audio to be generated in the stereo field.

13. The method according to any one of claims 3-10, characterized in that, The touch action includes multiple actions; the action of controlling the vehicle's audio system to output an audio signal in response to a user's touch action on a vehicle's cabin component includes: Determine the trigger time of the touch action; For multiple touch actions at the same time, the sub-audio signals corresponding to each touch action are synthesized to generate the audio signal and output.

14. The method according to any one of claims 3-10, characterized in that, The cockpit components corresponding to the touch action include multiple components; The method of controlling the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin includes: Based on the touch action corresponding to each cockpit component, determine the sub-audio signal corresponding to each cockpit component; The sub-audio signals are synthesized to obtain the audio signal, which is then output.

15. The method according to claim 14, characterized in that, The determination of the sub-audio signal corresponding to each cockpit component based on the touch action corresponding to each of the cockpit components includes: Determine the type of musical instrument corresponding to each of the aforementioned cockpit components; Based on the instrument type, a sub-audio signal corresponding to the striking action is determined so that multiple instruments can output the audio signal based on the same beat.

16. The method according to any one of claims 3-10, characterized in that, The touch action corresponds to multiple cockpit components; the step of controlling the vehicle's audio system to output an audio signal in response to the user's touch action on the vehicle's cockpit components includes: Based on the number of touch actions corresponding to each cockpit component within a preset time period, the volume parameters corresponding to the sub-audio signals of each cockpit component are adjusted to achieve volume balance.

17. The method according to any one of claims 3-10, characterized in that, The touch action corresponds to multiple cockpit components; the step of controlling the vehicle's audio system to output an audio signal in response to the user's touch action on the vehicle's cockpit components includes: Based on the target music style, determine the corresponding timbre parameters, reverb parameters, and / or delay parameters; The audio signal is generated and output based on the timbre parameters, the reverberation parameters, and / or the delay parameters.

18. The method according to any one of claims 3-10, characterized in that, The method of controlling the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin includes: If the difference between the output time of the guiding instruction and the trigger time of the touch action is less than a first preset duration threshold, the sound device is controlled to output a note corresponding to the guiding instruction, and the lighting device is controlled to output light feedback.

19. The method according to claim 18, characterized in that, The method of controlling the vehicle's audio system to output an audio signal in response to a user's touch action on a component of the vehicle's cabin also includes: If the difference between the trigger time of the touch action and the output time of the guidance instruction is greater than or equal to the first preset duration threshold and less than the second preset duration threshold, the output time of the audio signal corresponding to the touch action is corrected to the preset time, where the preset time is the closest musical score beat point to the trigger time.

20. The method according to any one of claims 3-10, characterized in that, The method of controlling the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin includes: If the actual interactive area corresponding to the touch action does not match the target interactive area corresponding to the guidance instruction, the erroneous note corresponding to the actual interactive area is replaced with the target note corresponding to the target interactive area, and / or a transition sound effect is output to replace the erroneous note.

21. The method according to any one of claims 3-10, characterized in that, The method of controlling the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin includes: If the number of consecutive correct taps by the user exceeds the preset number, perform at least one of the following operations: Output combo sound effects; Control the vehicle's lighting system to adjust the light color temperature; The audio device is controlled to add drum beats and / or bass lines to the audio signal.

22. The method according to any one of claims 3-10, characterized in that, The method of controlling the vehicle's audio system to output audio signals in response to a user's touch action on a component of the vehicle's cabin includes: Responding to a user's touch sequence on vehicle cabin components; Based on the touch action sequence, determine the note sequence corresponding to the touch action sequence; Based on the note sequence, the audio system of the vehicle is controlled to output audio signals.

23. The method according to any one of claims 1-22, characterized in that, The step of controlling the vehicle's audio system to output an audio signal in response to user interaction input includes: The interactive signals generated by the interactive input are acquired through onboard sensors. Feature extraction is performed on the interaction signal to determine the interaction features; the interaction features are used to characterize the signal quality of the interaction signal. If the interaction feature characterization meets the preset quality conditions, the interaction signal is determined to be a valid interaction signal. Based on the effective interaction signal, the audio device in the vehicle is controlled to output an audio signal.

24. The method according to claim 23, characterized in that, The method for acquiring the interaction signal includes at least one of the following: Acquired through the sensors corresponding to the steering wheel; Acquired through sensors corresponding to the inner panel and / or instrument panel; Acquired via sensors corresponding to armrests and / or seats and / or carpets; Acquired through the sensor corresponding to the brake pedal; Acquired via camera; Acquired via microphone.

25. The method according to claim 23, characterized in that, The interactive input is a touch action; the interactive signal is a tap signal; when the cockpit component is the steering wheel, the valid tap signal is determined in the following way: Based on a first condition, the tapping signals obtained by the sensor corresponding to the steering wheel are filtered to obtain candidate tapping signals; the first condition includes: the amplitude of the tapping signal is greater than a preset amplitude threshold, the rate of change of the amplitude of the tapping signal is greater than a preset rate threshold, and the duration of the tapping signal is less than a preset tapping time. The candidate tapping signals are filtered based on a second condition to obtain the valid tapping signals; the second condition includes: The target grip parameter is less than or equal to a preset grip parameter threshold; the target grip parameter includes: grip duration and / or grip contact area; The time difference between the triggering time of the tapping signal and the triggering time of the steering wheel button signal is less than or equal to a first time difference; The time difference between the triggering time of the knocking signal and the occurrence time of the vehicle vibration signal is less than or equal to the second time difference.

26. The method according to claim 24, characterized in that, The brake pedal corresponds to a first transmission link and a second transmission link; the first transmission link and the second transmission link are isolated from each other; The first transmission link is used to transmit the braking signal; the second transmission link is used to transmit the interaction signal.

27. The method according to claim 26, characterized in that, The input signal of the first transmission link includes the acquisition signal of the brake pedal sensor, and the interaction signal transmitted by the second transmission link is a copy of the acquisition signal in the first transmission link.

28. The method according to claim 24, characterized in that, The steering wheel corresponds to the third and fourth transmission links; the third and fourth transmission links are isolated from each other; The signal transmitted by the third transmission link is used for driver grip detection; the fourth transmission link is used to acquire and transmit the interaction signal.

29. The method according to claim 28, characterized in that, The input signal of the third transmission link includes the acquisition signal of the sensor corresponding to the steering wheel, and the interaction signal transmitted by the fourth transmission link is a copy of the acquisition signal in the third transmission link.

30. The method according to any one of claims 1-29, characterized in that, The step of controlling the vehicle's audio system to output an audio signal in response to user interaction input includes: When the vehicle meets the preset safety conditions, in response to the user's interactive input, the system controls the vehicle's audio device to output the audio signal; The preset security conditions include: The vehicle is in Park (P) gear. The vehicle's speed is zero; The vehicle is in the parking brake state; The vehicle's braking control function is not activated; The vehicle's steering control function was not in emergency takeover mode; The vehicle's active safety features were not activated.

31. The method according to claim 30, characterized in that, The method further includes: If the vehicle does not meet the preset safety conditions and the source of the interactive input is the driver's position, control the second transmission link corresponding to the brake pedal to not transmit the initial signal, and / or control the fourth transmission link corresponding to the steering wheel to not transmit the initial signal, and / or control the target function to be downgraded or turned off.

32. The method according to claim 30, characterized in that, The method further includes: If the vehicle does not meet the preset safety conditions, the source of the interactive input is a non-driver's position, and the touch action is applied to the steering wheel, when the vehicle speed is less than or equal to the preset vehicle speed, the target function is controlled to operate in a restricted mode; or, when the vehicle speed is greater than the preset vehicle speed, the target function is controlled to be downgraded or turned off.

33. The method according to any one of claims 2-32, characterized in that, The cockpit components include at least one of the following: steering wheel, interior panel, seat, dashboard, armrest, carpet, and brake pedal.

34. The method according to any one of claims 1-33, characterized in that, The interactive input includes at least one of the following: tapping, slapping, light touching, heavy hitting, continuous rhythmic movements, stomping movements, voice, gestures, or swiping.

35. The method according to any one of claims 1-34, characterized in that, The target function is a mode that generates corresponding audio signals based on user interactive input.

36. The method according to claim 35, characterized in that, The target function includes at least one of the following modes: Playing mode; the playing mode is used to indicate that cockpit components and / or interactive areas are mapped to different notes to simulate musical instrument playing; Ensemble mode; the ensemble mode is used to indicate that different cockpit components are mapped to different instrument types to simulate a multi-instrument ensemble; Sheet music guidance mode; the sheet music guidance mode is used to instruct and guide the user to perform different interactive inputs in order to play the sheet music.

37. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1-36.

38. A computer-readable storage medium, characterized in that, When the computer-executable instructions stored in the computer-readable storage medium are executed by the processor of the device, the device is capable of performing the method as described in any one of claims 1-36.

39. A vehicle, characterized in that, include: The electronic device of claim 37, and / or the computer-readable storage medium of claim 38.

40. A computer program product, the computer program product comprising computer instructions, characterized in that, When the computer instructions are executed on the processor of the device, the device is enabled to perform the method as described in any one of claims 1-36.