A method, system, device and medium for generating soundscape and vibration feedback of electric vehicle

Abstract

The present application discloses a method, system, device and medium for generating soundscape and vibration feedback of electric vehicles. The method includes: collecting vehicle condition information and driver physiological data information; performing feature fusion on the vehicle condition information and driver physiological data information to obtain fusion features; inputting the fusion features into a pre-established neural network model to generate control parameters; and according to the control parameters, controlling the synthesizer and the light to generate driving soundscape and provide real-time vibration feedback through a vibration motor. The present application adjusts the in-vehicle soundscape and generates vibration feedback according to the driver's physiological data and vehicle condition, creating a more relaxing driving environment, thereby having a positive impact on the driver and improving the comfort and safety of the electric vehicle.

Description

Technical Field

[0001] The present application relates to the field of automobile sound and vibration technology, and in particular to a method, system, device and medium for generating soundscape and vibration feedback of an electric vehicle. Background Art

[0002] In traditional internal combustion engine vehicles, the sound of the engine plays an important role in enhancing the driver's experience of vehicle power and helps improve comfort and safety. However, with the rapid development of electric vehicles, their quiet characteristics have brought new challenges to drivers and pedestrians. Electric vehicles lack the engine noise and vibration of traditional internal combustion engine vehicles, causing drivers to lose important auditory and tactile feedback, affecting the driving experience and safety. Therefore, to make up for this shortcoming, existing automakers are trying different sounds, from high-tech whistles to more traditional engine noises, but the several sounds currently designed are difficult to meet the various needs of customers in terms of variety. Secondly, the sound generated is not naturally integrated with the in-car environment; in addition, there is a lack of real-time vibration feedback, depriving the driver of important information about vehicle dynamics and characteristics, greatly affecting the driver's judgment of the vehicle's condition and bringing traffic risks to pedestrians. The above problems need to be solved. Summary of the invention

[0003] The main purpose of this application is to overcome the shortcomings and deficiencies of the prior art and to provide a method, system, device and medium for generating soundscape and vibration feedback of an electric vehicle, so as to adjust the in-vehicle soundscape and generate vibration feedback according to the driver's physiological data and the vehicle condition, thereby creating a more relaxing driving environment, thereby having a positive impact on the driver and improving the comfort and safety of the electric vehicle.

[0004] In order to achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, the present application provides a method for generating soundscape and vibration feedback of an electric vehicle, comprising the following steps:

[0006] Collect vehicle condition information and driver physiological data information;

[0007] Performing feature fusion on the vehicle condition information and the driver's physiological data information to obtain fusion features;

[0008] Inputting the fused features into a pre-established neural network model to generate control parameters;

[0009] According to the control parameters, the synthesizer and the lights are controlled to generate a driving soundscape and real-time vibration feedback is provided through a vibration motor.

[0010] As a preferred technical solution, the vehicle condition information includes: speed, pedal pressure and torque;

[0011] The driver's physiological data information includes: EEG, heart rate and facial expression.

[0012] As a preferred technical solution, the vehicle condition information and the driver's physiological data information are subjected to feature fusion, specifically:

[0013] Each single-value feature of the vehicle condition information and the heart rate feature and EEG feature in the physiological data information are expanded into a two-dimensional data form; the facial expression is a three-dimensional data form;

[0014] The two-dimensional data form and the three-dimensional data form are superimposed and fused on channels to obtain fusion features.

[0015] As a preferred technical solution, the neural network model includes: a convolutional layer, a pooling layer, a GRU layer and a fully connected layer.

[0016] As a preferred technical solution, the step of inputting the fusion features into a pre-established neural network model to generate control parameters includes:

[0017] The convolution layer extracts local features of the fused features to obtain a first feature;

[0018] The pooling layer downsamples the first feature to obtain a second feature;

[0019] The GRU layer performs time series signal analysis on the second feature to obtain a feature containing time series information;

[0020] The fully connected layer integrates and maps the features containing the timing information to generate control parameters.

[0021] As a preferred technical solution, a self-attention mechanism is introduced after the pooling layer to assign different weights to the second features to highlight important second features.

[0022] As a preferred technical solution, the control parameters include: synthesizer control parameters, lighting control parameters and vibration motor control parameters.

[0023] In a second aspect, the present application provides an electric vehicle soundscape and vibration feedback generation system, which is applied to the electric vehicle soundscape and vibration feedback generation method, including an information acquisition module, a feature fusion module, a control parameter module, and a soundscape and vibration feedback module;

[0024] The information collection module is used to collect vehicle condition information and driver physiological data information;

[0025] The feature fusion module is used to perform feature fusion on the vehicle condition information and the driver's physiological data information to obtain fusion features;

[0026] The control parameter module is used to input the fusion features into a pre-established neural network model to generate control parameters;

[0027] The soundscape and vibration feedback module is used to control the synthesizer and the light to generate the driving soundscape according to the control parameters and to provide real-time vibration feedback through the vibration motor.

[0028] In a third aspect, the present application provides an electronic device, the electronic device comprising:

[0029] at least one processor; and,

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the electric vehicle soundscape and vibration feedback generation method.

[0032] In a fourth aspect, the present application provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method for generating soundscape and vibration feedback of an electric vehicle.

[0033] In summary, compared with the prior art, the effective effects brought about by the technical solution provided by this application include at least:

[0034] The present application proposes a method for generating soundscape and vibration feedback of an electric vehicle, which generates control parameters in real time through vehicle condition information and driver physiological data, and generates personalized driving soundscape and vibration feedback according to the control parameters; the generated soundscape is made diverse and adaptive, and at the same time, the naturalness of the synthesized soundscape is enhanced through vibration tactile feedback, thereby also improving the driver's driving experience, comfort and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flow chart of a method for generating soundscape and vibration feedback of an electric vehicle provided in one embodiment of the present application;

[0037] Figure 2 A schematic diagram of a neural network model structure provided for one embodiment of the present application;

[0038] Figure 3 A schematic diagram of the module structure of a synthesizer provided in one embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of a sound effect module provided for one embodiment of the present application;

[0040] Figure 5 A block diagram of a soundscape and vibration feedback generation system for an electric vehicle provided in one embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0042] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0043] Example:

[0044] See also Figure 1 In one embodiment of the present application, a method for generating soundscape and vibration feedback of an electric vehicle is provided, comprising the following steps:

[0045] S1. Collect vehicle condition information and driver physiological data information.

[0046] Furthermore, the vehicle condition information includes other vehicle conditions such as speed, pedal pressure and torque; the driver's physiological data information includes: EEG, heart rate and facial expressions, etc.

[0047] S2. Perform feature fusion on the vehicle condition information and the driver's physiological data information to obtain fusion features.

[0048] Furthermore, step S2 is to fuse the collected vehicle condition information and driver physiological data information into a three-dimensional data form for subsequent analysis and processing of the neural network model. The fusion steps include:

[0049] The three-dimensional data form N*M*C can be preset, where N is the length, M is the width, and C is the number of channels.

[0050] S21. Duplicate each single-value feature (speed, pedal pressure, torque, etc.) of the vehicle condition information and expand these single-value features into a two-dimensional data form N*M.

[0051] S22. For the EEG signal features in the driver's physiological data information, the time-frequency transformation method is used to convert it from the original time domain signal to the frequency domain signal to form a two-dimensional data form; the heart rate feature is in a single-valued form, and the replication method is also used to expand it into a two-dimensional data form; and the driver's facial expression is a single three-channel color photo collected by a camera, which itself is a three-dimensional data form N*M*3.

[0052] S23, superimposing and fusing each two-dimensional data form and three-dimensional data form in the channel dimension to obtain a fusion feature.

[0053] Specifically, the two-dimensional data forms of vehicle speed, pedal pressure, torque, etc. are first superimposed on the channel dimension to form a new two-dimensional fusion feature; then, the two-dimensional data form of the EEG signal after time-frequency transformation is also superimposed on this new two-dimensional fusion feature; then, the two-dimensional data form of the heart rate is also superimposed on this fusion feature; finally, the three-channel color photo of the facial expression is superimposed on the previous fusion feature to form an overall three-dimensional fusion feature N*M*C.

[0054] The above-mentioned fusion feature method fully considers the characteristics and dimensions of various types of data, ensuring the consistency and integrity of the data during the fusion process; at the same time, through the superposition fusion on the channel, it effectively combines information from different sources together, providing more rich and comprehensive input data for the subsequent deep learning neural network model.

[0055] S3, inputting the fusion features into a pre-established deep learning neural network model to generate control parameters, the specific steps include:

[0056] For further information, see Figure 2 ,The neural network model includes an input layer, a convolutional layer, a pooling layer, a GRU (Gated Recurrent Unit) layer, a fully connected layer, and a final output layer.

[0057] S31, the input layer receives the fused features, and the convolution layer extracts the local features of the fused features (including vehicle conditions and driver physiological data information) to obtain the first features.

[0058] S32: The pooling layer downsamples the first feature to reduce the data dimension and obtain the second feature.

[0059] S33, the GRU layer performs time series signal analysis on the second feature to obtain a feature containing time series information.

[0060] Furthermore, since physiological characteristics and vehicle condition characteristics are analyzed and processed in real time, the GRU layer is introduced to analyze the time series signals to provide useful information for subsequent decision-making and control. GRU is an improvement on LSTM, which simplifies the entire structure operation and enhances performance.

[0061] S34, the fully connected layer integrates and maps the features containing the timing information, generates control parameters, and outputs them through the output layer.

[0062] The control parameters include synthesizer control parameters, light control parameters and vibration motor control parameters.

[0063] Furthermore, after the convolutional layer and the pooling layer, a self-attention mechanism is introduced. The self-attention mechanism allows the model to automatically learn and selectively focus on important information in the input, and assign different weights to the second features, thereby highlighting important feature information and improving the performance and generalization ability of the model.

[0064] In this embodiment, the training data of the deep learning neural network model comes from the laboratory simulated driving scene. First, the synthesizer control parameters, lighting control parameters and vibration control parameters of the vibration motor corresponding to most driving conditions (vehicle conditions and driver physiology) are preset based on experience, and then the parameters are fine-tuned according to the driver's own style and comfort in the simulated driving experiment; the adjusted data is used as the training data of the neural network model.

[0065] Since it is a simulation of the real-time driving process, the driver needs to express his driving experience at regular intervals. An experienced synthesizer tuner can set the synthesizer and lighting control parameters in real time based on the driver's feedback.

[0066] S4. According to the control parameters, control the synthesizer and the light to generate a personalized driving soundscape and provide real-time vibration feedback through the vibration motor.

[0067] The synthesizer in this embodiment includes a sound source module, a pitch module, an envelope module, an equalizer module, a sound effect module, a gain module and a speaker. Figure 3 .

[0068] The sound source module: The sound source includes a real fuel vehicle sound library and basic square waves, triangle waves, white noise, pink noise, etc.

[0069] The pitch module: The pitch module raises or lowers the pitch of the sound source and controls the pitch to be raised or lowered through a neural network model.

[0070] The envelope module: The envelope module contains 4 parameters, namely Attack, Decay, Sustain, and Release. Attack: This section determines the length of time required for the sound to reach the initial maximum volume from the beginning. Decay: The length of time that the sound decays immediately after reaching the maximum volume. The volume after decay is the volume maintained later. Sustain: The duration of the constant volume in the sustain stage. Release: This is the final stage of the sound, representing the length of time it takes for the sound to gradually decay from the maintained volume to the 0 level (minimum volume); the size of its 4 values ​​can be controlled by the neural network model.

[0071] The equalizer module: The equalizer is composed of three types of filters: low shelf, high shelf, and peak. The parameters controlled by the neural network model include cutoff frequency, center frequency, and gain.

[0072] The sound effect module: The sound effects include reverb, flanger, phase shift, tremolo and other effects, please refer to Figure 4 ; The parameter controlled by the neural network model is the dry-wet ratio of each sound effect.

[0073] The gain module: The gain controls the volume of the final synthesizer output, and the neural network model controls its gain parameters.

[0074] The vibration motor control parameters of this embodiment are mainly the vibration amplitude and vibration type, which are controlled by the output of the neural network model. The vibration motor is installed on the main driver's seat in the cab to achieve sound-touch coordination in a tactile way. The vibration signal is collected from fuel vehicles under different vehicle conditions, filtered and analyzed, and then stored locally as an offline signal. In the actual driving process of the electric vehicle in this application, the parameters of the deep learning neural network model are used to select and call the offline vibration signal stored locally. Vibration feedback enhances the naturalness of the synthesized sound and provides the driver with a comfortable internal experience.

[0075] In this embodiment, the lighting information includes lighting lamps and instrument panel lighting displays, and the main control parameters are brightness and color temperature.

[0076] In summary, the present application proposes a method for generating electric vehicle soundscape and vibration feedback. Since soundscape is closely related to emotions, it is necessary to collect the driver's physiological data information, and then fuse the driver's physiological data with the vehicle condition information, use a deep learning neural model to identify the driver's overall emotions from multimodal data such as facial expressions, heart rate, EEG activity, and vehicle condition information, and generate a variety of sounds and vibration feedback through a synthesizer to further enhance the driver's perception of the naturalness of the synthesized sound and the perception of the vehicle condition; therefore, the present application can provide a personalized driving soundscape suitable for the driver's overall emotions in real time, improving the driver's driving experience, comfort, and safety.

[0077] It should be noted that, for the sake of convenience of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously.

[0078] Based on the same idea as the electric vehicle soundscape and vibration feedback generation method in the above embodiment, the present application also provides an electric vehicle soundscape and vibration feedback generation system, which can be used to execute the above electric vehicle soundscape and vibration feedback generation method. For ease of explanation, the structural schematic diagram of an electric vehicle soundscape and vibration feedback generation system embodiment only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0079] See also Figure 5 In another embodiment of the present application, a soundscape and vibration feedback generation system 100 for an electric vehicle is provided, the system comprising an information acquisition module 101, a feature fusion module 102, a control parameter module 103, and a soundscape and vibration feedback module 104;

[0080] The information collection module 101 is used to collect vehicle condition information and driver physiological data information;

[0081] The feature fusion module 102 is used to perform feature fusion on the vehicle condition information and the driver's physiological data information to obtain fusion features;

[0082] The control parameter module 103 is used to input the fusion features into a pre-established neural network model to generate control parameters;

[0083] The soundscape and vibration feedback module 104 is used to control the synthesizer and the light to generate the driving soundscape according to the control parameters and to provide real-time vibration feedback through the vibration motor.

[0084] It should be noted that an electric vehicle soundscape and vibration feedback generation system of the present application corresponds one-to-one to an electric vehicle soundscape and vibration feedback generation method of the present application. The technical features and beneficial effects described in the embodiment of the above-mentioned electric vehicle soundscape and vibration feedback generation method are applicable to the embodiment of an electric vehicle soundscape and vibration feedback generation system. For specific contents, please refer to the description in the embodiment of the method of the present application, which will not be repeated here. This is hereby declared.

[0085] In addition, in the implementation of the electric vehicle soundscape and vibration feedback generation system of the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the electric vehicle soundscape and vibration feedback generation system is divided into different program modules to complete all or part of the functions described above.

[0086] In another embodiment, an electronic device for a method for generating soundscape and vibration feedback of an electric vehicle is provided, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, a method for generating soundscape and vibration feedback of an electric vehicle according to any embodiment of the present application is implemented.

[0087] Exemplarily, in this embodiment, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present application. The one or more module elements may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the device;

[0088] The device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The device may include, but is not limited to, a processor, a memory;

[0089] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device, and uses various interfaces and lines to connect various parts of the entire device;

[0090] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; in addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0091] Accordingly, the present application also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a method for generating soundscape and vibration feedback of an electric vehicle as described in any one of the above embodiments.

[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The above embodiments are preferred implementation modes of the present application, but the implementation modes of the present application are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present application should be equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. A method for generating soundscape and vibration feedback of an electric vehicle, characterized in that: The steps include: Collect vehicle condition information and driver physiological data information; The vehicle condition information includes: speed, pedal pressure and torque; the driver's physiological data information includes: EEG, heart rate and facial expression; The vehicle condition information and the driver's physiological data information are subjected to feature fusion to obtain fusion features; the vehicle condition information and the driver's physiological data information are subjected to feature fusion, specifically: Each single-value feature of the vehicle condition information and the heart rate feature and EEG feature in the physiological data information are expanded into a two-dimensional data form; the facial expression is a three-dimensional data form; Superimpose and fuse the two-dimensional data form and the three-dimensional data form on channels to obtain fusion features; Inputting the fused features into a pre-established neural network model to generate control parameters; According to the control parameters, the synthesizer and the lights are controlled to generate driving sound scenes and real-time vibration feedback is provided through the vibration motor; the control parameters include: synthesizer control parameters, lighting control parameters and vibration motor control parameters; the vibration motor control parameters include vibration amplitude and vibration type; the vibration motor is installed on the main driving seat of the cab to achieve sound-touch coordination in a tactile manner.

2. The method for generating soundscape and vibration feedback of an electric vehicle according to claim 1, characterized in that: The neural network model includes: a convolutional layer, a pooling layer, a GRU layer and a fully connected layer.

3. The method for generating soundscape and vibration feedback of an electric vehicle according to claim 2, characterized in that: The step of inputting the fusion features into a pre-established neural network model to generate control parameters comprises: The convolution layer extracts local features of the fused features to obtain a first feature; The pooling layer downsamples the first feature to obtain a second feature; The GRU layer performs time series signal analysis on the second feature to obtain a feature containing time series information; The fully connected layer integrates and maps the features containing the timing information to generate control parameters.

4. The method for generating soundscape and vibration feedback of an electric vehicle according to claim 3, characterized in that: It also includes introducing a self-attention mechanism after the pooling layer to assign different weights to the second features to highlight important second features.

5. A soundscape and vibration feedback generation system for an electric vehicle, characterized in that: A method for generating soundscape and vibration feedback of an electric vehicle applied to any one of claims 1 to 4, comprising an information acquisition module, a feature fusion module, a control parameter module, and a soundscape and vibration feedback module; The information collection module is used to collect vehicle condition information and driver physiological data information; The vehicle condition information includes: speed, pedal pressure and torque; the driver's physiological data information includes: EEG, heart rate and facial expression; The feature fusion module is used to perform feature fusion on the vehicle condition information and the driver's physiological data information to obtain fusion features; the feature fusion on the vehicle condition information and the driver's physiological data information is specifically: Each single-value feature of the vehicle condition information and the heart rate feature and EEG feature in the physiological data information are expanded into a two-dimensional data form; the facial expression is a three-dimensional data form; Superimpose and fuse the two-dimensional data form and the three-dimensional data form on channels to obtain fusion features; The control parameter module is used to input the fusion features into a pre-established neural network model to generate control parameters; The soundscape and vibration feedback module is used to control the synthesizer and the light to generate the driving soundscape and provide real-time vibration feedback through the vibration motor according to the control parameters; the control parameters include: synthesizer control parameters, light control parameters and vibration motor control parameters; the vibration motor control parameters include vibration amplitude and vibration type; the vibration motor is installed on the main driving seat of the cab to achieve sound-touch coordination in a tactile manner.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for generating soundscape and vibration feedback of an electric vehicle as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for generating soundscape and vibration feedback of an electric vehicle as described in any one of claims 1 to 4 is implemented.

Images