4d seat rhythm multi-modal perception fusion regulation method and device

Abstract

The application discloses a 4D seat rhythm multi-modal perception fusion regulation method and device, relates to the technical field of intelligent cockpit, and comprises the following steps: acquiring a cockpit audio signal, a vehicle running state signal and a passenger physiological and posture signal of a target vehicle, obtaining a structured multi-modal original data packet, performing standardization processing, and obtaining a multi-dimensional feature vector set; performing global space-time synchronous alignment processing based on the multi-dimensional feature vector set, and cooperating with a preset fusion weight to obtain a fusion feature vector; based on the fusion feature vector, obtaining a multi-system collaborative control instruction, and performing rhythm control on a cockpit collaborative system of the target vehicle. Through distributed acquisition of multi-modal original data, adaptive standardization feature extraction, AI-driven global space-time synchronous alignment and dynamic weight fusion, the application realizes accurate collaborative control of the cockpit multi-system, improves seat rhythm perception accuracy, enhances scene adaptive capability, and eliminates the technical effect of the phase difference of multi-system signals.

Description

Technical Field

[0001] This application relates to the field of intelligent cockpit technology, specifically to a 4D seat rhythm multimodal perception fusion control method and device. Background Technology

[0002] With the rapid development of intelligent cockpit technology in new energy vehicles, 4D dynamic seats have become an important feature for enhancing the driving and riding experience. Existing 4D dynamic seat rhythm control technology is mainly based on a single audio signal drive, which has problems such as limited perception dimension, poor spatiotemporal synchronization, and insufficient adaptive capability, making it difficult to meet users' needs for an immersive and personalized driving and riding experience.

[0003] The existing technology has the following main technical defects: (1) Existing technologies rely solely on a single audio signal for seat rhythm control, resulting in low rhythm perception accuracy and insufficient adaptive adjustment capability.

[0004] (2) Existing technologies suffer from poor spatiotemporal synchronization of multi-source heterogeneous data and large system delay fluctuations, resulting in a phase difference between the actual vibration of the seat and the target signal.

[0005] (3) Existing technologies have not achieved coordinated control of seat rhythm, in-vehicle audio, and vehicle driving status, resulting in poor consistency of experience. Therefore, in order to address the above problems and meet practical needs, a 4D seat rhythm multimodal perception fusion control technology is proposed. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application aims to provide a 4D seat rhythm multimodal perception fusion and control method and device. Using an in-vehicle intelligent cockpit domain controller as the core hardware carrier, it achieves precise collaborative control of multiple cockpit systems through distributed acquisition of multimodal raw data, adaptive standardized feature extraction, AI-driven global spatiotemporal synchronization alignment, and dynamic weight fusion. Each level is sequentially connected according to the data processing link to form a closed-loop feedback, ultimately achieving the technical effects of improving seat rhythm perception accuracy, enhancing scene adaptability, and eliminating phase differences between multi-system signals.

[0007] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a 4D seat rhythm multimodal perception fusion and control method, the method comprising the following steps: Acquire the cockpit audio signal, vehicle operating status signal, and occupant physiological and posture signal of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. The structured multimodal raw data packet is standardized to obtain the corresponding multidimensional feature vector set; Global spatiotemporal synchronization alignment is performed based on the multi-dimensional feature vector set, and corresponding fused feature vectors are obtained by combining the preset fusion weights. Based on the fused feature vector, rhythmic control is performed on the cockpit collaboration system of the target vehicle; The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

[0008] Based on the above technical solution, the step of acquiring the cockpit audio signal, vehicle operating status signal, and occupant physiological and posture signals of the target vehicle, and synchronizing them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp, includes the following steps: The original digital audio signal of the target vehicle and the background noise signal of the cabin environment are acquired, and noise reduction processing is performed to obtain the cabin audio signal. The vehicle speed, lateral acceleration, longitudinal acceleration, steering angle, and road bump signals of the target vehicle are collected and integrated to obtain the vehicle's operating status signal. The occupant's sitting posture information, weight distribution information, heart rate and respiratory rate are collected and obtained from the target vehicle, and the occupant's physiological and posture signals are integrated. Based on a unified clock, the cockpit audio signal, the vehicle operating status signal, and the occupant physiological and posture signals are synchronized to obtain the structured multimodal raw data packet.

[0009] Based on the above technical solution, the standardization process of the structured multimodal raw data packet to obtain the corresponding multidimensional feature vector set includes the following steps: The cockpit audio signal is standardized to obtain the corresponding audio time-domain feature vector, audio frequency-domain feature vector and audio semantic feature vector, and then integrated to obtain the cockpit audio feature vector. The vehicle operating status signal is subjected to outlier removal and sliding smoothing to obtain the vehicle operating status feature vector. The occupant physiological and posture signals are uniformly quantized and encoded to obtain occupant physiological and posture feature vectors. The multi-dimensional feature vector set is obtained by integrating the cockpit audio feature vector, the vehicle operating status feature vector, and the occupant physiological and posture feature vector.

[0010] Based on the above technical solution, the step of performing global spatiotemporal synchronization alignment processing based on the multi-dimensional feature vector set, and obtaining the corresponding fused feature vector with preset fusion weights, includes the following steps: Based on the cockpit audio feature vector, vehicle operating status feature vector, and occupant physiological and posture feature vector in the multi-dimensional feature vector set, global spatiotemporal synchronization alignment is performed, and with the preset fusion weight, a fused feature vector is obtained.

[0011] Based on the above technical solution, the step of performing rhythmic control on the cockpit collaboration system of the target vehicle based on the fused feature vector includes the following steps: Based on the fused feature vector, a set of seat rhythm control instructions and cockpit multi-system collaborative control instructions are generated, and integrated to obtain multi-system collaborative control instructions.

[0012] Secondly, this application provides a 4D seat rhythm multimodal perception fusion control device, the device comprising: The multimodal data acquisition module is used to acquire the cockpit audio signal, vehicle operating status signal and occupant physiological and posture signal of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. The data standardization processing module is used to standardize the structured multimodal raw data packet to obtain the corresponding multidimensional feature vector set; The feature vector fusion module is used to perform global spatiotemporal synchronization alignment processing based on the multi-dimensional feature vector set, and obtain the corresponding fused feature vector with preset fusion weights. The instruction output control module is used to perform rhythmic control on the cockpit collaboration system of the target vehicle based on the fused feature vector. The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

[0013] Based on the above technical solution, the multimodal data acquisition module is also used to collect the original digital audio signal of the target vehicle and the background noise signal of the cabin environment, perform noise reduction processing, and obtain the cabin audio signal. The multimodal data acquisition module is also used to collect the vehicle speed, lateral acceleration, longitudinal acceleration, steering angle and road bump signals of the target vehicle, and integrate them to obtain the vehicle operating status signal. The multimodal data acquisition module is also used to collect and obtain the occupant sitting posture information, weight distribution information, heart rate and respiratory rate corresponding to the occupants of the target vehicle, and integrate them to obtain the occupant physiological and posture signals. The multimodal data acquisition module is also used to synchronize the cockpit audio signal, the vehicle operating status signal, and the occupant physiological and posture signals based on a unified clock to obtain the structured multimodal raw data packet.

[0014] Based on the above technical solution, the data standardization processing module is also used to standardize the cockpit audio signal to obtain the corresponding audio time domain feature vector, audio frequency domain feature vector and audio semantic feature vector, and integrate them to obtain the cockpit audio feature vector. The data standardization processing module is also used to perform outlier removal and sliding smoothing on the vehicle operating status signal to obtain the vehicle operating status feature vector. The data standardization processing module is also used to perform unified quantization encoding on the occupant physiological and posture signals to obtain occupant physiological and posture feature vectors. The data standardization processing module is also used to integrate the cockpit audio feature vector, the vehicle operating status feature vector, and the occupant physiological and posture feature vector to obtain the multi-dimensional feature vector set.

[0015] Based on the above technical solution, the feature vector fusion module is also used to perform global spatiotemporal synchronization alignment based on the cockpit audio feature vector, vehicle operating status feature vector and occupant physiological and posture feature vector in the multi-dimensional feature vector set, and to fuse them with preset fusion weights to obtain a fused feature vector.

[0016] Based on the above technical solution, the instruction output control module is also used to generate a seat rhythm control instruction set and a cockpit multi-system collaborative control instruction based on the fused feature vector, and integrate them to obtain the multi-system collaborative control instruction.

[0017] Compared with the prior art, the advantages of this application are: This application uses an in-vehicle intelligent cockpit domain controller as the core hardware carrier. Through distributed acquisition of multimodal raw data, adaptive standardized feature extraction, AI-driven global spatiotemporal synchronization alignment, and dynamic weight fusion, it achieves precise collaborative control of multiple cockpit systems. Each level is sequentially connected according to the data processing link to form a closed-loop feedback, ultimately achieving the technical effects of improving the accuracy of seat rhythm perception, enhancing scene adaptation capabilities, and eliminating signal phase differences among multiple systems. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of the 4D seat rhythm multimodal perception fusion and control method according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the perception layer architecture principle of the 4D seat rhythm multimodal perception fusion and control method according to an embodiment of this application. Figure 3 This is a schematic diagram of the feature layer architecture principle of the 4D seat rhythm multimodal perception fusion and control method according to an embodiment of this application; Figure 4 This is a schematic diagram of the fusion layer processing flow of the 4D seat rhythm multimodal perception fusion and control method according to an embodiment of this application; Figure 5 This is a structural block diagram of the 4D seat rhythm multimodal perception fusion control device according to an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0022] This application provides a 4D seat rhythm multimodal perception fusion control method and device, which uses an in-vehicle intelligent cockpit domain controller as the core hardware carrier. Through distributed acquisition of multimodal raw data, adaptive standardized feature extraction, AI-driven global spatiotemporal synchronization alignment and dynamic weight fusion, it achieves precise collaborative control of multiple cockpit systems. Each level is sequentially connected according to the data processing link to form a closed-loop feedback, ultimately achieving the technical effects of improving seat rhythm perception accuracy, enhancing scene adaptability, and eliminating phase differences between multi-system signals.

[0023] To achieve the aforementioned technical effects, the overall concept of this application is as follows: A 4D seat rhythm multimodal perception fusion and control method, the method includes the following steps: S1. Acquire the cockpit audio signal, vehicle operating status signal, and occupant physiological and posture signals of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. S2. Standardize the original structured multimodal data packets to obtain the corresponding multidimensional feature vector set; S3. Perform global spatiotemporal synchronization alignment based on a multi-dimensional feature vector set, and obtain the corresponding fusion feature vector with preset fusion weights; S4. Based on the fused feature vector, rhythmic control is performed on the cockpit collaboration system of the target vehicle; The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

[0024] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0025] Firstly, see [the following] Figures 1-4 As shown in the figure, this application provides a 4D seat rhythm multimodal perception fusion control method, which includes the following steps: S1. Acquire the cockpit audio signal, vehicle operating status signal, and occupant physiological and posture signals of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. S2. Standardize the original structured multimodal data packets to obtain the corresponding multidimensional feature vector set; S3. Perform global spatiotemporal synchronization alignment based on a multi-dimensional feature vector set, and obtain the corresponding fusion feature vector with preset fusion weights; S4. Based on the fused feature vector, rhythmic control is performed on the cockpit collaboration system of the target vehicle; The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

[0026] In this embodiment, an in-vehicle intelligent cockpit domain controller is used as the core hardware carrier. Through distributed acquisition of multimodal raw data, adaptive standardized feature extraction, AI-driven global spatiotemporal synchronization alignment and dynamic weight fusion, precise collaborative control of multiple cockpit systems is achieved. Each level is sequentially connected according to the data processing link to form a closed-loop feedback, ultimately achieving the technical effects of improving the accuracy of seat rhythm perception, enhancing scene adaptation capabilities, and eliminating phase differences between signals from multiple systems.

[0027] Furthermore, the acquisition of the target vehicle's cabin audio signals, vehicle operating status signals, and occupant physiological and posture signals, and the synchronization based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp, includes the following steps: The original digital audio signal of the target vehicle and the background noise signal of the cabin environment are acquired, and noise reduction processing is performed to obtain the cabin audio signal. The vehicle speed, lateral acceleration, longitudinal acceleration, steering angle, and road bump signals of the target vehicle are collected and integrated to obtain the vehicle's operating status signal. The occupant's sitting posture information, weight distribution information, heart rate and respiratory rate are collected and obtained from the target vehicle, and the occupant's physiological and posture signals are integrated. Based on a unified clock, the cockpit audio signal, the vehicle operating status signal, and the occupant physiological and posture signals are synchronized to obtain the structured multimodal raw data packet.

[0028] Furthermore, the standardization process for the structured multimodal raw data packet to obtain the corresponding multidimensional feature vector set includes the following steps: The cockpit audio signal is standardized to obtain the corresponding audio time-domain feature vector, audio frequency-domain feature vector and audio semantic feature vector, and then integrated to obtain the cockpit audio feature vector. The vehicle operating status signal is subjected to outlier removal and sliding smoothing to obtain the vehicle operating status feature vector. The occupant physiological and posture signals are uniformly quantized and encoded to obtain occupant physiological and posture feature vectors. The multi-dimensional feature vector set is obtained by integrating the cockpit audio feature vector, the vehicle operating status feature vector, and the occupant physiological and posture feature vector.

[0029] Furthermore, the step of performing global spatiotemporal synchronization alignment processing based on the multi-dimensional feature vector set, and obtaining the corresponding fused feature vector with preset fusion weights, includes the following steps: Based on the cockpit audio feature vector, vehicle operating status feature vector, and occupant physiological and posture feature vector in the multi-dimensional feature vector set, global spatiotemporal synchronization alignment is performed, and with the preset fusion weight, a fused feature vector is obtained.

[0030] Furthermore, the step of performing rhythmic control on the cockpit collaboration system of the target vehicle based on the fused feature vector includes the following steps: Based on the fused feature vector, a set of seat rhythm control instructions and cockpit multi-system collaborative control instructions are generated, and integrated to obtain multi-system collaborative control instructions.

[0031] Based on the technical solution of this application embodiment, a five-layer edge-coordinated 4D dynamic seat rhythm multimodal perception fusion system based on the perception layer, feature layer, fusion layer, control layer, and execution layer can be constructed, as detailed below: First, the perception layer: Inputs: cockpit audio signals, vehicle operating status signals, and occupant physiological and posture signals.

[0032] Output: Structured multimodal raw data packets with uniform timestamps.

[0033] Core processing logic: Adopting a distributed perception architecture, under the unified clock synchronization scheduling of the vehicle intelligent cockpit domain controller, it realizes parallel acquisition and preliminary encapsulation of multi-source data, providing a benchmark for subsequent spatiotemporal alignment.

[0034] Specific components and data flow: Audio acquisition unit: Acquires the raw digital audio signal from the in-vehicle audio playback system, and simultaneously acquires the background noise signal of the cabin environment for subsequent noise reduction processing of audio features. Vehicle status perception unit: Collects vehicle operating status data such as vehicle speed, longitudinal / lateral acceleration, steering angle, and road bumpiness. Occupant Status Sensing Unit: Collects physiological and postural data such as occupant sitting posture, weight distribution, heart rate, and respiratory rate.

[0035] Preferred implementation: Each unit in the perception layer establishes a low-latency communication link with the domain controller via the vehicle-mounted Ethernet. All acquired data is timestamped at the acquisition end based on the domain controller's global clock, and the timestamp accuracy is synchronized with the domain controller's clock. The perception layer architecture principle is illustrated in the attached diagram of the specification. Figure 2 As shown.

[0036] Second, the feature layer: Input: The original multimodal data packets with timestamps output from the perception layer.

[0037] Output: A standardized, fusionable set of multi-dimensional feature vectors.

[0038] Core processing logic: Based on the inherent characteristics of different modal data, differentiated preprocessing and feature extraction methods are adopted, and adaptive normalization and quantization of features are achieved to eliminate dimensional differences and distribution biases between different modal data.

[0039] Specific components and data flow: The multi-scale audio feature extraction module performs frame-by-frame preprocessing on the original audio signal and extracts time-domain, frequency-domain, and high-level semantic features in parallel. The time-domain features include rhythm points, beat intensity, and energy envelope; the frequency-domain features include spectral distribution and harmonic components; and the semantic features include song style, mood type, and paragraph division (verse, chorus, bridge). Furthermore, this module adopts an adaptive scaling mechanism to automatically adjust the time window and frequency resolution of feature extraction according to the dynamic range of the audio signal, thereby improving the robustness of feature extraction. Vehicle status feature normalization module: performs outlier removal and sliding smoothing on vehicle operation status data, and uses a standardization method to map features of different dimensions to a unified numerical range. Occupant status feature quantization module: It performs unified quantization encoding on discrete features such as occupant sitting posture and weight distribution and continuous features such as heart rate and respiratory rate to generate feature vectors that represent occupant's physical preferences and physiological state.

[0040] Preferred implementation: The feature layer adopts a parallel computing architecture to achieve synchronous extraction of multimodal features. Each feature extraction submodule runs independently, and data processing results are exchanged through shared memory. The principle of the feature layer architecture is shown in the attached diagram of the specification. Figure 3 As shown.

[0041] Third, the fusion layer: Input: A set of multi-dimensional feature vectors output by the feature layer.

[0042] Output: A spatiotemporally aligned, dynamically weighted fused feature vector.

[0043] Core processing logic: Based on AI adaptive algorithms, first, global spatiotemporal synchronization and alignment of multi-source heterogeneous data is completed to eliminate phase difference. Then, the fusion weights of each modality feature are dynamically adjusted according to scene and state characteristics. Finally, scene adaptive correction is performed to generate the optimal fusion feature.

[0044] Specific components and data flow: Multimodal spatiotemporal synchronization alignment module: Using the timing of the seat rhythm signal as a global benchmark, it employs a deep learning-based timing alignment algorithm to perform global timing calibration on the control signals of all cabin collaborative systems, such as audio, lighting, air conditioning, and aromatherapy. Furthermore, this module adopts a sliding window prediction mechanism to monitor system delay fluctuations in real time and predict signal timing deviations in advance, performing pre-compensation to reduce the impact of delay fluctuations. Dynamic weight fusion algorithm module: Constructs a fusion network based on attention mechanism, and automatically assigns weights to each modality feature in the fusion process according to audio semantic features, vehicle state features and occupant state features; Scene Adaptive Correction Module: Based on a pre-trained scene classification model, it identifies the current cabin usage scenario (such as music appreciation, audio-visual entertainment, and relaxation), and makes targeted corrections to the fused features according to the characteristics of the scenario to optimize the output effect of seat rhythm.

[0045] Preferred implementation: The fusion layer is deployed on the edge computing unit of the in-vehicle intelligent cockpit domain controller to achieve local real-time data processing; the fusion layer processing flow diagram is shown in the attached figure. Figure 4 As shown.

[0046] Fourth, the control layer Input: The fused feature vector output by the fusion layer.

[0047] Output: Seat rhythm control command set and cabin multi-system collaborative control commands after feedforward compensation and arbitration.

[0048] Core processing logic: Based on fusion features, multi-degree-of-freedom rhythmic commands for the seat are generated. Command errors are corrected through motor dynamic response feedforward compensation. Simultaneously, a collaborative control arbitration module enables coordinated scheduling of seat rhythms with systems such as audio, lighting, air conditioning, and aromatherapy. This section focuses on real-time command generation and multi-system collaborative arbitration; specific module functions and implementation logic are not elaborated here.

[0049] Fifth, the execution layer: Input: The set of control instructions output by the control layer.

[0050] Output: Multi-degree-of-freedom vibration motion of the seat and synchronous execution effect of multiple systems in the cockpit.

[0051] Core processing logic: Receive control commands and parse them into drive signals for each execution component, drive the corresponding component to perform actions according to the synchronous timing sequence, and feed back the execution status to the control layer to form a closed-loop control.

[0052] Specific components and data flow: Seat multi-degree-of-freedom actuator assembly: including seat cushion vibration motor, backrest vibration motor, lumbar support adjustment motor, and headrest vibration motor, which respectively realize independent vibration and posture adjustment of different parts of the seat; In-vehicle audio playback module: Plays audio signals according to the synchronized timing sequence.

[0053] Preferred implementation: Each component of the execution layer adopts a distributed drive architecture, with each actuator independently receiving drive signals and performing actions, thereby improving the overall system response speed.

[0054] Sixth, system data flow: Each unit in the perception layer collects multimodal raw data in parallel and adds a timestamp based on the global clock of the domain controller to each data frame, which is then transmitted to the vehicle intelligent cockpit domain controller via the vehicle Ethernet. The data first enters the feature layer to complete the preprocessing of multimodal data and standardized feature extraction; The extracted feature vectors are input into the fusion layer, and then undergo multimodal spatiotemporal synchronization alignment, dynamic weight fusion, and scene adaptive correction to generate fused feature vectors. The fused feature vectors are transmitted to the control layer to generate seat rhythm control commands and cockpit multi-system collaborative control commands, and to complete the feedforward compensation and arbitration of the commands; The final control command is sent to the execution layer, where it is executed synchronously by the seat multi-degree-of-freedom actuator component and the in-vehicle audio playback module. At the same time, the execution layer feeds back the execution status of each component to the control layer, forming a complete closed-loop control link.

[0055] In summary, the technical advantages of the technical solutions in this application are as follows: (1) Compared with the shortcomings of existing technologies that rely on a single audio signal for seat rhythm control and have a single perception dimension, the technical solution of this application integrates a multimodal perception architecture of an audio acquisition unit, a vehicle state perception unit and an occupant state perception unit. It combines a multi-scale audio feature extraction method with a dynamic weight fusion algorithm based on fuzzy hierarchical analysis to achieve collaborative perception and dynamic adjustment of audio, vehicle state and occupant state data. This effectively solves the problem that "existing technologies rely on a single audio signal for seat rhythm control, resulting in low rhythm perception accuracy and insufficient adaptive adjustment capability".

[0056] (2) Compared with the shortcomings of the prior art, such as poor spatiotemporal synchronization of multi-source heterogeneous data and motor response delay causing a phase difference between seat vibration and target signal, the technical solution of this application uses a two-level spatiotemporal synchronization mechanism of "global clock hard synchronization + cross-correlation soft calibration" to control the synchronization error of multi-source data within ±5ms. Combined with the phase and amplitude feedforward compensation of the second-order linear dynamic response model of the seat actuator, the motor response delay is controlled within 10ms, realizing accurate spatiotemporal alignment of multi-modal signals and seat vibration, effectively solving the problem of "poor spatiotemporal synchronization of multi-source heterogeneous data and large system delay fluctuation in the prior art, resulting in a phase difference between actual seat vibration and target signal".

[0057] (3) Compared with the shortcomings of existing fusion algorithms, such as insufficient generalization and inability to adapt to individual differences among passengers and complex driving scenarios, the technical solution of this application divides four typical scenarios by combining the scene recognition sub-module of audio type and vehicle status and presets basic weight range. It introduces passenger weight, sitting posture and heart rate as weight real-time correction factors and combines the online incremental learning mechanism of user habits to enable the fusion algorithm to adapt to passengers with different weights and sitting postures as well as complex driving scenarios such as music appreciation and movie watching, effectively solving the problem that "existing fusion algorithms have insufficient generalization and cannot adapt to individual differences among passengers and complex driving scenarios".

[0058] (4) Compared with the shortcomings of the prior art, which has not achieved multi-system collaborative control, experience consistency and safety, the technical solution of this application establishes a three-level collaborative control mechanism of seat rhythm and in-vehicle audio playback, vehicle driving safety and passenger physiological state by setting up an independent collaborative control arbitration module. This achieves precise synchronization of auditory and tactile feedback, dynamic adjustment with driving safety priority and intelligent adaptation of passenger comfort, effectively solving the problem that "the prior art has not achieved collaborative control of seat rhythm and in-vehicle audio and vehicle driving state, resulting in poor experience consistency".

[0059] (5) Compared with the shortcomings of the prior art in lacking personalized customization capabilities, the technical solution of this application supports users to customize the amplitude, frequency and mode of seat rhythm, and gradually optimizes the fusion weight by recording user usage habits through the online learning module, thereby realizing personalized customization of seat rhythm for each user and significantly improving the user's immersive driving experience.

[0060] Secondly, see Figure 5 As shown in the figure, this application provides a 4D seat rhythm multimodal perception fusion control device, which includes: The multimodal data acquisition module is used to acquire the cockpit audio signal, vehicle operating status signal and occupant physiological and posture signal of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. The data standardization processing module is used to standardize the structured multimodal raw data packet to obtain the corresponding multidimensional feature vector set; The feature vector fusion module is used to perform global spatiotemporal synchronization alignment processing based on the multi-dimensional feature vector set, and obtain the corresponding fused feature vector with preset fusion weights. The instruction output control module is used to perform rhythmic control on the cockpit collaboration system of the target vehicle based on the fused feature vector. The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

[0061] In this embodiment, an in-vehicle intelligent cockpit domain controller is used as the core hardware carrier. Through distributed acquisition of multimodal raw data, adaptive standardized feature extraction, AI-driven global spatiotemporal synchronization alignment and dynamic weight fusion, precise collaborative control of multiple cockpit systems is achieved. Each level is sequentially connected according to the data processing link to form a closed-loop feedback, ultimately achieving the technical effects of improving the accuracy of seat rhythm perception, enhancing scene adaptation capabilities, and eliminating phase differences between signals from multiple systems.

[0062] Furthermore, the multimodal data acquisition module is also used to collect the original digital audio signal of the target vehicle and the background noise signal of the cabin environment, perform noise reduction processing, and obtain the cabin audio signal. The multimodal data acquisition module is also used to collect the vehicle speed, lateral acceleration, longitudinal acceleration, steering angle and road bump signals of the target vehicle, and integrate them to obtain the vehicle operating status signal. The multimodal data acquisition module is also used to collect and obtain the occupant sitting posture information, weight distribution information, heart rate and respiratory rate corresponding to the occupants of the target vehicle, and integrate them to obtain the occupant physiological and posture signals. The multimodal data acquisition module is also used to synchronize the cockpit audio signal, the vehicle operating status signal, and the occupant physiological and posture signals based on a unified clock to obtain the structured multimodal raw data packet.

[0063] Furthermore, the data standardization processing module is also used to standardize the cockpit audio signal to obtain the corresponding audio time-domain feature vector, audio frequency-domain feature vector and audio semantic feature vector, and integrate them to obtain the cockpit audio feature vector. The data standardization processing module is also used to perform outlier removal and sliding smoothing on the vehicle operating status signal to obtain the vehicle operating status feature vector. The data standardization processing module is also used to perform unified quantization encoding on the occupant physiological and posture signals to obtain occupant physiological and posture feature vectors. The data standardization processing module is also used to integrate the cockpit audio feature vector, the vehicle operating status feature vector, and the occupant physiological and posture feature vector to obtain the multi-dimensional feature vector set.

[0064] Furthermore, the feature vector fusion module is also used to perform global spatiotemporal synchronization alignment based on the cockpit audio feature vector, vehicle operating status feature vector, and occupant physiological and posture feature vector in the multi-dimensional feature vector set, and to fuse them with preset fusion weights to obtain a fused feature vector.

[0065] Furthermore, the instruction output control module is also used to generate a seat rhythm control instruction set and a cockpit multi-system collaborative control instruction based on the fused feature vector, and integrate them to obtain the multi-system collaborative control instruction.

[0066] In summary, the 4D seat rhythm multimodal perception fusion control device provided in this application embodiment has the same technical principle as the 4D seat rhythm multimodal perception fusion control method provided in the first aspect in terms of technical problems, technical solutions and technical effects, so it will not be described in detail here.

[0067] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the system or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0068] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A 4D seat rhythm multimodal perception fusion and control method, characterized in that, The method includes the following steps: Acquire the cockpit audio signal, vehicle operating status signal, and occupant physiological and posture signal of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. The structured multimodal raw data packet is standardized to obtain the corresponding multidimensional feature vector set; Global spatiotemporal synchronization alignment is performed based on the multi-dimensional feature vector set, and corresponding fused feature vectors are obtained by combining the preset fusion weights. Based on the fused feature vector, the corresponding multi-system collaborative control command is obtained, and the cockpit collaborative system of the target vehicle is rhythmically controlled. The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

2. The 4D seat rhythm multimodal perception fusion and control method as described in claim 1, characterized in that, The process of acquiring the target vehicle's cabin audio signals, vehicle operating status signals, and occupant physiological and posture signals, and synchronizing them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp, includes the following steps: The original digital audio signal of the target vehicle and the background noise signal of the cabin environment are acquired, and noise reduction processing is performed to obtain the cabin audio signal. The vehicle speed, lateral acceleration, longitudinal acceleration, steering angle, and road bump signals of the target vehicle are collected and integrated to obtain the vehicle's operating status signal. The occupant's sitting posture information, weight distribution information, heart rate and respiratory rate are collected and obtained from the target vehicle, and the occupant's physiological and posture signals are integrated. Based on a unified clock, the cockpit audio signal, the vehicle operating status signal, and the occupant physiological and posture signals are synchronized to obtain the structured multimodal raw data packet.

3. The 4D seat rhythm multimodal perception fusion and control method as described in claim 1, characterized in that, The standardization process for the structured multimodal raw data packet to obtain the corresponding multidimensional feature vector set includes the following steps: The cockpit audio signal is standardized to obtain the corresponding audio time-domain feature vector, audio frequency-domain feature vector and audio semantic feature vector, and then integrated to obtain the cockpit audio feature vector. The vehicle operating status signal is subjected to outlier removal and sliding smoothing to obtain the vehicle operating status feature vector. The occupant physiological and posture signals are uniformly quantized and encoded to obtain occupant physiological and posture feature vectors. The multi-dimensional feature vector set is obtained by integrating the cockpit audio feature vector, the vehicle operating status feature vector, and the occupant physiological and posture feature vector.

4. The 4D seat rhythm multimodal perception fusion and control method as described in claim 1, characterized in that, The global spatiotemporal synchronization alignment process based on the multi-dimensional feature vector set, combined with preset fusion weights, to obtain the corresponding fused feature vector includes the following steps: Based on the cockpit audio feature vector, vehicle operating status feature vector, and occupant physiological and posture feature vector in the multi-dimensional feature vector set, global spatiotemporal synchronization alignment is performed, and with the preset fusion weight, a fused feature vector is obtained.

5. The 4D seat rhythm multimodal perception fusion and control method as described in claim 1, characterized in that, The step of performing rhythmic control on the cockpit collaboration system of the target vehicle based on the fused feature vector includes the following steps: Based on the fused feature vector, a set of seat rhythm control instructions and cockpit multi-system collaborative control instructions are generated, and integrated to obtain multi-system collaborative control instructions.

6. A 4D seat rhythm multimodal perception fusion control device, characterized in that, The device includes: The multimodal data acquisition module is used to acquire the cockpit audio signal, vehicle operating status signal and occupant physiological and posture signal of the target vehicle, and synchronize them based on a unified clock to obtain a structured multimodal raw data packet with a unified timestamp. The data standardization processing module is used to standardize the structured multimodal raw data packet to obtain the corresponding multidimensional feature vector set; The feature vector fusion module is used to perform global spatiotemporal synchronization alignment processing based on the multi-dimensional feature vector set, and obtain the corresponding fused feature vector with preset fusion weights. The instruction output control module is used to perform rhythmic control on the cockpit collaboration system of the target vehicle based on the fused feature vector. The cockpit collaboration system includes at least an audio system, a lighting system, an air conditioning system, and an aromatherapy system.

7. The 4D seat rhythm multimodal perception fusion and control device as described in claim 6, characterized in that: The multimodal data acquisition module is also used to collect the original digital audio signal of the target vehicle and the background noise signal of the cabin environment, perform noise reduction processing, and obtain the cabin audio signal. The multimodal data acquisition module is also used to collect the vehicle speed, lateral acceleration, longitudinal acceleration, steering angle and road bump signals of the target vehicle, and integrate them to obtain the vehicle operating status signal. The multimodal data acquisition module is also used to collect and obtain the occupant sitting posture information, weight distribution information, heart rate and respiratory rate corresponding to the occupants of the target vehicle, and integrate them to obtain the occupant physiological and posture signals. The multimodal data acquisition module is also used to synchronize the cockpit audio signal, the vehicle operating status signal, and the occupant physiological and posture signals based on a unified clock to obtain the structured multimodal raw data packet.

8. The 4D seat rhythm multimodal perception fusion control device as described in claim 6, characterized in that: The data standardization processing module is also used to standardize the cockpit audio signal to obtain the corresponding audio time-domain feature vector, audio frequency-domain feature vector and audio semantic feature vector, and integrate them to obtain the cockpit audio feature vector. The data standardization processing module is also used to perform outlier removal and sliding smoothing on the vehicle operating status signal to obtain the vehicle operating status feature vector. The data standardization processing module is also used to perform unified quantization encoding on the occupant physiological and posture signals to obtain occupant physiological and posture feature vectors. The data standardization processing module is also used to integrate the cockpit audio feature vector, the vehicle operating status feature vector, and the occupant physiological and posture feature vector to obtain the multi-dimensional feature vector set.

9. The 4D seat rhythm multimodal perception fusion and control device as described in claim 6, characterized in that: The feature vector fusion module is also used to perform global spatiotemporal synchronization alignment based on the cockpit audio feature vector, vehicle operating status feature vector, and occupant physiological and posture feature vector in the multi-dimensional feature vector set, and to fuse them with preset fusion weights to obtain a fused feature vector.

10. The 4D seat rhythm multimodal perception fusion control device as described in claim 6, characterized in that: The command output control module is also used to generate a set of seat rhythm control commands and cockpit multi-system collaborative control commands based on the fused feature vector, and integrate them to obtain multi-system collaborative control commands.

Images