A Vehicle Head-Up Display Method and System for Hearing-Impaired Drivers

Abstract

The present invention belongs to the technical field of intelligent driving, and specifically relates to a vehicle head-up display method and system for hearing-impaired drivers. It includes the following steps: S1, obtaining sound signals for complex driving environments; S2, constructing a voice recognition model for hearing-impaired drivers; S3, constructing a vehicle head-up display system for hearing-impaired drivers; S4, the driver assists in driving according to the information displayed by the vehicle head-up display system. The present invention enables hearing-impaired drivers to better perceive the surrounding driving environment, improves driving safety, and solves the problem that hearing-impaired drivers cannot perceive the sound information of the surrounding driving environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent driving, and specifically relates to a vehicle head-up display method and system for hearing-impaired drivers. Background Art

[0002] With the progress of society and the continuous improvement of civilization, ensuring the travel rights of disabled people has received extensive attention from society. As a special group of drivers, hearing-impaired drivers cannot accurately and effectively hear the sound information of the surrounding driving environment, such as horn sounds and sirens of special vehicles, which even become an important factor affecting safe travel. For drivers with low hearing disabilities, they can still wear hearing aids or cochlear implants and other assistive hearing devices. However, severely hearing-impaired drivers cannot hear any sounds at all and cannot perceive and understand the surrounding driving environment. Especially in case of emergencies, they are likely to miss important sound signals, posing severe challenges to driving safety and convenience.

[0003] Vehicle head-up display technology can display key traffic information in front of the driver's field of vision, effectively reducing the number of times the driver looks down at the instrument panel, achieving effective human-machine interaction, and improving driving safety and comfort. However, traditional vehicle head-up display technology still cannot solve the problem of the lack of hearing-dimensional information for hearing-impaired drivers. With the continuous development of large models such as speech recognition and machine translation, the sound signal data of the driving environment obtained by the vehicle's sound sensors can be converted into text information and displayed on the in-vehicle display screen, enabling hearing-impaired drivers to comprehensively perceive the surrounding driving environment. In addition, complex head-up display information also brings a heavy burden to drivers. Therefore, combining speech recognition, machine translation with head-up display technology to develop a head-up display system for hearing-impaired drivers is not only a need for technological innovation but also an important manifestation of social responsibility. It is of great significance for improving driving safety, providing an autonomous and comfortable driving experience for hearing-impaired drivers, and providing important support for realizing a more equal and safe travel environment and promoting the comprehensive development of intelligent transportation systems.

[0004] In the field of intelligent cockpits and human-machine interaction, there are relatively few patents on vehicle head-up display systems and methods for hearing-impaired drivers. Chinese Patent CN 119065139 A discloses a head-up display control system and a display method for a head-up display, which projects key information onto the windshield, reduces the driver's line-of-sight transfer during driving, and improves driving efficiency. Chinese Patent CN 110531551 A discloses a liquid crystal display device and a head-up display system, which solves the problem of heat generation caused by long-term irradiation of the display module and can enhance the image brightness. Chinese Patent CN 206773941 U discloses a vehicle head-up display device for people with red-green color blindness and color weakness, which converts the traffic light image information into a signal that can be recognized by red-green color-blind drivers and projects it onto the windshield of the vehicle. The above three patents can transmit driving information to the windshield and reduce the driver's perception load. However, due to the lack of sound signals, they still cannot meet the needs of hearing-impaired drivers. Therefore, it is still very crucial to develop a head-up display system and method for hearing-impaired drivers. Summary of the Invention

[0005] The present invention provides a vehicle head-up display method for hearing-impaired drivers, enabling hearing-impaired drivers to better perceive the surrounding driving environment, improving driving safety, and solving the problem that hearing-impaired drivers cannot perceive the sound information of the surrounding driving environment.

[0006] The technical solution of the present invention is described in conjunction with the accompanying drawings as follows:

[0007] In a first aspect, the present invention provides a vehicle head-up display method for hearing-impaired drivers, including the following steps:

[0008] S1. Obtain sound signals for complex driving environments;

[0009] S2. Construct a sound recognition model for hearing-impaired drivers;

[0010] S3. Construct a vehicle head-up display system for hearing-impaired drivers.

[0011] Further, the specific method of S1 is as follows:

[0012] S11. Obtain sound signals based on sensors;

[0013] S12. Obtain driving sound signals of web crawlers;

[0014] S13. Obtain sound signals based on generative adversarial networks;

[0015] Further, the specific method of S11 is as follows:

[0016] S111. Arrange vehicle sound sensors;

[0017] Install a sound sensor on each side of the vehicle head to collect sound signals from the front and sides; install a sound sensor on each side of the vehicle tail to obtain the vehicle horn sounds from the rear and the side-rear.

[0018] S112. Define the acquisition method;

[0019] According to the Nyquist sampling theorem, set the sampling rate to twice the highest frequency of the signal, and all frequency sound signals have the same sensitivity; use multiple channels and ensure that the sound sensors have the same sensitivity to sound signals in different directions.

[0020] S113. Digitally process the sound signals;

[0021] Sample the analog signal at fixed time intervals to discretize the continuous signal; convert the amplitude value of each sampling point into a digital value with finite precision; convert the quantized digital value into a binary code, and the coding method is PCM.

[0022] S114. Compress and store the data of the sound signals;

[0023] Adopt the method of lossless compression and local storage.

[0024] Furthermore, the specific method of S12 is as follows:

[0025] S121. Grab sound data based on keywords;

[0026] Use web crawler technology to grab driving sound signal data from the Internet based on keywords; first, determine the target website and data source, analyze the HTML structure of the target web page, and locate the position of the target data; extract the links, titles, descriptions, and tags of the target data from the search result page.

[0027] S122. Download sound data based on multiple threads;

[0028] Use the method of thread pool to implement multiple threads; the sound signal web page parsing module and the download module each maintain a thread pool; according to the remaining capacity of the thread pool, allocate a new thread. When the web page parsing task or the download task is completed, destroy the thread, and at this time the current remaining capacity is incremented by 1; when it is detected that the remaining capacity of the thread pool is 0, wait for the parsing task or the download task to complete and release the resources, and then allocate resources to the new thread.

[0029] S123. Store and classify the sound signals;

[0030] Store the metadata information of the audio file in a database or a file; convert the downloaded audio file into a unified format; classify the audio files and label each segment according to the timestamp; finally, upload the data to the cloud storage service.

[0031] Furthermore, the specific method of S13 is as follows:

[0032] S131. Preprocess the sound signal;

[0033] Use a noise reduction algorithm to remove the environmental noise in the audio files of S11 and S12; divide the driving scene sound signal into short-time frames, perform Fourier transform on each frame of the signal to obtain the spectrum; subtract the noise spectrum from the spectrum of the noisy signal, and finally convert the processed spectrum back to the time-domain signal; then perform standardization or normalization on the noise-reduced sound data and scale the waveform data to between [-1, 1];

[0034] S132. Construct a sound generation model;

[0035] The sound generation model includes a generator and a discriminator. Use a one-dimensional transposed convolution to construct the generator and a one-dimensional convolution to construct the discriminator; the input of the generator is a random noise vector and the output is the generated sound waveform data; the generator includes multiple fully connected layers or convolutional layers; first, map the random noise data through the fully connected layer to a middle dimension; use the deconvolution layer to gradually upsample the data to the dimension of the target waveform data; the output layer outputs the generated sound waveform data with the same dimension as the real data; the input of the discriminator is the real driving sound signal data or the fake data generated by the generator, and the output is a scalar representing the probability that the input data is real data; use the binary cross-entropy loss function to alternately train the generator and the discriminator, so that the discriminator can correctly distinguish real data and fake data, and the fake data generated by the generator can deceive the discriminator into thinking that the fake data is real, and finally generate data of the same type as the real driving sound waveform data;

[0036] S133. Perform reliability analysis on the sound signal data generated in S132;

[0037] Post-process the generated sound signal; use objective indicators such as signal-to-noise ratio and logarithmic spectrum distance to evaluate the quality of the generated signal. The higher the signal-to-noise ratio, the better the signal quality, and the smaller the logarithmic spectrum distance, the closer the generated signal is to the target signal; and evaluate the quality of the generated signal through listening tests to ensure that the timbre, rhythm, and dynamic range of the driving sound signal conform to the auditory characteristics of the real scene;

[0038] S134. Integrate the sound signal data for complex driving scenes;

[0039] The reliability driving sound data generated by integrating S11, S12, and S132 L , L =( L 1, L 2,..., L i ,..., L N ) for a total of N pieces of driving sound signal data. Among them, each piece of sound waveform signal data L i has a different data length.

[0040] Furthermore, the specific method of S2 is as follows:

[0041] S21. Construct a driving scenario sound signal data set;

[0042] S22. Construct a driving sound recognition model that fuses time-frequency domain features;

[0043] S23. Conduct performance test and evaluation on the driving sound recognition model.

[0044] Furthermore, the specific method of S21 is as follows:

[0045] S211. Extract data features;

[0046] Extract features from the equal-length sound waveform data integrated by S134, including time-domain feature extraction and frequency-domain feature extraction; the time-domain features include mean, variance, and peak value; use the fast Fourier transform to convert the time-domain sound signal into a frequency-domain sound signal, expressed as a two-dimensional array of frequency and time, calculate the power spectrum from the complex matrix of the Fourier transform, and perform matrix multiplication on the power spectrum and the Mel filter bank to obtain the Mel spectrogram, where the horizontal axis is time and the vertical axis is Mel frequency;

[0047] S212. Label the data;

[0048] Label the collected driving sound data; classify and organize the driving sound data according to the scenario and sound category, and assign one or more category labels to the sound in each time period;

[0049] S213. Divide the training set, test set, and validation set;

[0050] Divide the driving sound data into equal-length sound wavelength data according to the sliding window method. The sliding window is w , and the sliding step is s ; the calculation formula for the number of samples in each sound data is shown in (1):

[0051] (1)

[0052] Wherein, is the length of the i-th sound data sample;

[0053] N The calculation formula (2) of the total number of samples of the driving sound signal data is:

[0054] (2)

[0055] Finally, the samples are divided into a training set, a test set, and a validation set according to a ratio.

[0056] Furthermore, the specific method of S22 is as follows:

[0057] S221. Temporal feature representation learning based on the attention mechanism;

[0058] Extract time series features from the temporal waveform signal; Sound waveform signal data with timestamps X ={ x 1, x 2,…, x T}, where x t is the waveform signal at time t , T is the time step; The original sound waveform signal , The waveform signal after position encoding is expressed as formula (3):

[0059] (3)

[0060] Extract features through the encoder, and the encoder consists of multiple layers of multi-head self-attention mechanisms and feed-forward neural networks, and output the feature representation result H t , ;

[0061] (4)

[0062] S222. Frequency domain feature representation learning based on CNN;

[0063] Perform a short-time Fourier transform on X to obtain a spectrogram , where F is the frequency dimension, is the number of time frames; Obtain the intermediate feature representation result P through multiple layers of 2D convolution and pooling operations, and the calculation formula (5) is:

[0064] (5)

[0065] Finally, the frequency-domain feature representation result is obtained through dimensionality transformation and flattening operations. H f , ;

[0066] S223. Integrate the features based on the cross-attention mechanism;

[0067] Through the cross-attention mechanism, the time-domain feature representation result obtained in S221 H t , and the frequency-domain feature representation result of S222 H f are integrated to obtain the time-frequency domain integrated representation result H fused , and the calculation formula (6) is:

[0068] (6)

[0069] The calculation formula (7) of CrossAttention is:

[0070] (7)

[0071] Through the learnable weight matrix W Q , for H t perform a linear transformation to generate the corresponding Q vector. Through the weight matrices W K and W V perform a linear transformation on H f to generate the corresponding K , V vector;

[0072] S224. Generate the text for recognizing driving signals;

[0073] Through the transformer decoder, generate the word probability distribution through linear projection and Softmax; the decoder consists of l layers, including self-attention, cross-attention, and feed-forward networks; for the l th layer of the decoder, the input is the output of the previous layer, and the initial input is the embedding of the target sequence Y = y 1, y 2, …, y (t-1) , plus the positional encoding;

[0074] For the processing of the self-attention sublayer, the decoding result Zs is obtained through residual connection and layer normalization. The calculation formula (8) of Zs is:

[0075] (8)

[0076] Where , , , W Qs , W Ks and W Vs are learnable parameters;

[0077] For the processing of the cross-attention sublayer, the decoding result Zc is obtained. The calculation formula (9) of Zc is:

[0078] (9)

[0079] Where , , , W Qc , W Kc and W Vc are learnable parameters;

[0080] Then, through the feed-forward network sublayer, the calculation formula (10) is:

[0081] (10)

[0082] After residual connection and layer normalization, the representation result of the l th layer is obtained. The calculation formula (11) is:

[0083] (11)

[0084] Generate the word probability distribution through linear projection and Softmax. The calculation formula (12) is:

[0085] (12)

[0086] Finally, the text output of the voice signal is realized.

[0087] Furthermore, the specific method of S23 is as follows:

[0088] S231. Conduct offline test and evaluation on the model;

[0089] The mean absolute error, root mean square error, and mean absolute percentage error are used as evaluation indicators for the driving sound recognition model to evaluate the difference between the predicted value and the true value of the sound signal recognition model. The calculation formulas (13), (14), and (15) of MAE, RMSE, and MAPE are as follows:

[0090] (13)

[0091] (14)

[0092] (15)

[0093] In the formula, and are the true sound signal text and the predicted sound signal text, respectively;

[0094] S232. Conduct in-loop testing and evaluation on the model;

[0095] Deploy the trained model to in-vehicle hardware; ensure that the model can process sound data in real time when running on the hardware and provide timely feedback; and verify the real-time recognition ability of the model for key sounds in a real driving environment.

[0096] Furthermore, the specific method of S3 is as follows:

[0097] S31. Define the vehicle head-up display system architecture;

[0098] S32. Obtain vehicle head-up display information;

[0099] S33. Design a vehicle head-up display strategy for hearing-impaired drivers.

[0100] Furthermore, the specific method of S31 is as follows:

[0101] S311. Design system indicators and parameters;

[0102] Define head-up display system indicators, including the resolution and contrast of the displayed information, horizontal and vertical field of view angles, brightness, and chromaticity uniformity;

[0103] S312. Compose the head-up display system;

[0104] The head-up display system includes a MEMS scanning mirror, a laser, a laser diode driver, and a projection screen;

[0105] S313. Generate MEMS laser projection imaging;

[0106] Combining microelectromechanical systems and laser scanning technology, using red, green, and blue primary color lasers as light sources, collimating, focusing, and modulating the lasers through optical elements; after the laser beam passes through the MEMS micro-vibrating mirror, it is scanned according to the pixel information of the image data, and finally a complete image is formed on the screen.

[0107] Further, the specific method of S32 is as follows:

[0108] S321. Convert the sound signal into text information;

[0109] Deploy the driving sound recognition model that combines time-frequency domain features constructed in S22 on the vehicle hardware. The driving sound signal data obtained through the in-vehicle sound sensor is input into the vehicle controller VCU through the CAN bus, and after processing, the corresponding text information is output;

[0110] S322. Obtain the vehicle driving state information;

[0111] Obtain the acceleration and angular velocity through the inertial navigation instrument, and perform integral processing to obtain the speed and position information; monitor the voltage, current, and temperature of the battery through the battery management system to obtain the battery capacity and cruising range;

[0112] S323. Generate vehicle collision warning information;

[0113] The original data of the target object obtained through lidar, millimeter-wave radar, ultrasonic radar, and camera is processed and fused, combined with the vehicle's own speed and acceleration, to calculate the relative distance d and relative speed v rel , and predict the collision time ttc , ttc = d / v rel , and generate vehicle collision warning information;

[0114] S324. Integrate the vehicle head-up display information;

[0115] Integrate the sound signal text information, driving state information, and vehicle collision warning information output by S321, S322, and S323; the VCU sends the information to the head-up display system controller according to a fixed data format and message format. The head-up display system controller receives the signal, converts the received signal into a visual graph, and projects it on the screen through the optical projection system.

[0116] Further, the specific method of S33 is as follows:

[0117] S331. Design the head-up display interface;

[0118] Project the display information to different positions on the display screen according to different priorities; take the driving sound signal as the first priority and project it to the center of the screen where the driver is looking. The display information of the second priority is shown below the center of the screen, and the display information of the fourth priority and the third priority are shown on the left and right sides respectively.

[0119] S332. Determine the priority of the vehicle head-up display information;

[0120] Put the sound signal information that hearing-impaired drivers cannot obtain in the first priority to help drivers understand the driving environment around them; the display information of the second priority includes collision warning information, vehicle battery power shortage prompt information, lane departure information, and fault information; the third priority displays navigation information, including lane information, lane line information, traffic light information, and traffic sign information; the fourth priority displays external environment interaction information, including phone call display information, weather information, and in-vehicle temperature information.

[0121] In a second aspect, the present invention also provides a vehicle head-up display system for hearing-impaired drivers, which is used for the vehicle head-up display method for hearing-impaired drivers, and includes:

[0122] An acquisition module, which is used to acquire sound signals for complex driving environments;

[0123] A first construction module, which is used to construct a voice recognition model for hearing-impaired drivers;

[0124] A second construction module, which is used to construct a vehicle head-up display system for hearing-impaired drivers.

[0125] The beneficial effects of the present invention are:

[0126] 1. The vehicle head-up display method and system based on hearing-impaired drivers of the present invention include three steps: acquiring sound signals for complex driving environments, constructing a voice recognition model for hearing-impaired drivers, and constructing a vehicle head-up display system for hearing-impaired drivers; acquiring sound signals for complex driving environments, obtaining driving sound signal data through sensors and crawler methods, and generating vehicle driving sound signal data according to the generative adversarial network for data enhancement

[0127] 2. The vehicle head-up display method and system for hearing-impaired drivers of the present invention construct a voice recognition model for hearing-impaired drivers and recognize voice signals into text information;

[0128] 3. The vehicle head-up display method and system for hearing-impaired drivers according to the present invention design a head-up display interface and display text information in the interface in real time, realizing a friendly interaction between humans and machines, enabling hearing-impaired drivers to better perceive the surrounding driving environment and improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0129] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0130] Figure 1 is a schematic flowchart of the present invention;

[0131] Figure 2 is a schematic overall architecture diagram of step S1 in the present invention;

[0132] Figure 3 is a schematic overall architecture diagram of step S2 in the present invention;

[0133] Figure 4 is a schematic overall architecture diagram of step S3 in the present invention;

[0134] Figure 5 is an example diagram of the driving sound recognition model algorithm for fusing time-frequency domain features in S22 of the present invention;

[0135] Figure 6 is a schematic structural diagram of the hardware platform built in S22 of the present invention;

[0136] Figure 7 is an example diagram of the priority of the vehicle head-up display human-machine interaction interface in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0137] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the drawings rather than all structures.

[0138] Embodiment 1:

[0139] Refer to Figures 1-7 , this embodiment provides a vehicle head-up display method for hearing-impaired drivers, including the following steps:

[0140] S1. Obtain sound signals for complex driving environments, specifically as follows:

[0141] S11. Obtain a sensor-based sound signal;

[0142] S111. Arrange vehicle sound sensors;

[0143] Install a sound sensor on each side of the vehicle's front end to collect sound signals from the front and sides, such as the horn sounds of oncoming vehicles and the sounds of driving tires. Install a sound sensor on each side of the vehicle's rear end to obtain the horn sounds of vehicles behind and at the rear sides, such as the siren sounds of special vehicles like ambulances, police cars, and fire trucks;

[0144] S112. Define the acquisition method;

[0145] By setting a high signal-to-noise ratio, a wide dynamic range, and appropriate directivity, accurately capture the target sound signal; according to the Nyquist sampling theorem, set the sampling rate to twice the highest frequency of the signal, and all frequency sound signals have the same sensitivity; use multiple channels and ensure that the sound sensor has the same sensitivity to sound signals in different directions;

[0146] S113. Digitally process the sound signal;

[0147] Sample the analog signal at fixed time intervals to discretize the continuous signal; convert the amplitude value of each sampling point into a digital value with finite precision to ensure the dynamic range and precision of the signal; convert the quantized digital value into a binary code for easy storage and transmission, and use the PCM encoding method;

[0148] S114. Compress and store the data of the sound signal;

[0149] Use lossless compression to completely retain the original sound quality and avoid feature loss caused by compression. Use the local storage method to ensure adaptation to scenarios with high data access speed, low latency, and high real-time requirements.

[0150] S12. Obtain the driving sound signal of the web crawler;

[0151] S121. Crawl sound data based on keywords;

[0152] Use web crawler technology to crawl driving sound signal data from the Internet based on keywords; first, determine the target website and data source, and select an appropriate crawler framework and programming language according to the complexity of the target website; analyze the HTML structure of the target web page and locate the position of the target data; extract information such as the link, title, description, and tags of the target data from the search result page;

[0153] S122. Download sound data based on multiple threads;

[0154] Implement multi-threading using a thread pool to improve the efficiency of the crawler; the voice signal web parsing module and the download module each maintain a thread pool; allocate a new thread according to the remaining capacity of the thread pool, destroy the thread when the web parsing task or download task is completed, and at this time the current remaining capacity is incremented by 1; when it is detected that the remaining capacity of the thread pool is 0, wait for the parsing task or download task to complete and then release the resources, and then allocate resources to the new thread;

[0155] S123. Store and classify the voice signals;

[0156] Store the metadata information of the audio file (such as file name, download time, source website, etc.) in a database or file for convenient subsequent management and analysis; convert the downloaded audio files into a unified format; classify the audio files, and label each segment according to the time stamp, such as the siren of a characteristic vehicle, the horn sounds of vehicles in the front and side, the horn sounds of vehicles in the rear and side rear, etc. Finally, upload the data to the cloud storage service.

[0157] S13. Obtain the voice signals based on the adversarial generation network;

[0158] S131. Preprocess the voice signals;

[0159] Use a noise reduction algorithm to remove the environmental noise in the S11 and S12 audio files to improve the signal quality; divide the driving scene voice signal into short-time frames, perform Fourier transform on each frame of the signal to obtain the spectrum; subtract the noise spectrum from the spectrum of the noisy signal, and finally convert the processed spectrum back to the time-domain signal; then perform standardization or normalization processing on the noise-reduced voice data, and scale the waveform data to between [-1, 1];

[0160] S132. Construct a voice generation model;

[0161] The voice generation model includes a generator and a discriminator. Use one-dimensional transposed convolution to construct the generator and one-dimensional convolution to construct the discriminator; the input of the generator is a random noise vector, and the output is the generated voice waveform data; the generator includes multiple fully connected layers or convolutional layers; first, map the random noise data through the fully connected layer to a middle dimension; use the deconvolution layer to gradually upsample the data to the dimension of the target waveform data; the output layer outputs the generated voice waveform data, with the same dimension as the real data; the input of the discriminator is the real driving voice signal data or the fake data generated by the generator, and the output is a scalar representing the probability that the input data is real data; use the binary cross-entropy loss function to alternately train the generator and the discriminator, so that the discriminator can correctly distinguish between real data and fake data, and the fake data generated by the generator can deceive the discriminator into thinking that the fake data is real, and finally generate data identical to the real driving voice waveform data;

[0162] S133. Perform reliability analysis on the sound signal data generated in S132 to verify the validity of the data.

[0163] Perform post-processing on the generated sound signal (such as noise reduction, smoothing, etc.) to further improve the signal quality; use objective indicators such as signal-to-noise ratio and logarithmic spectral distance to evaluate the quality of the generated signal; and evaluate the quality of the generated signal through listening tests to ensure that the timbre, rhythm, and dynamic range of the driving sound signal conform to the auditory characteristics of the real scene.

[0164] S134. Integrate the sound signal data for complex driving scenarios.

[0165] Integrate the reliable driving sound data generated in S11, S12, and S132 L , L =( L 1, L 2,..., L i ,..., L N ) for a total of N pieces of driving sound signal data, where each sound waveform signal data L i has a different data length.

[0166] S2. Construct a sound recognition model for hearing-impaired drivers as follows:

[0167] S21. Construct a sound signal data set for driving scenarios.

[0168] S211. Extract data features.

[0169] Extract features from the equal-length sound waveform data integrated in S134, including time-domain feature extraction and frequency-domain feature extraction; the time-domain features include mean, variance, and peak value; use the fast Fourier transform to convert the time-domain sound signal into a frequency-domain sound signal, expressed as a two-dimensional array of frequency and time, calculate the power spectrum from the complex matrix of the Fourier transform, and perform matrix multiplication on the power spectrum and the Mel filter bank to obtain the Mel spectrogram, where the horizontal axis is time and the vertical axis is the Mel frequency.

[0170] S212. Label the data.

[0171] Label the collected driving sound data; classify and organize the driving sound data according to scenarios and sound categories, and assign one or more category labels to the sound in each time period; for example, the sirens of special vehicles in the surrounding environment, the honking of vehicles behind and on the side and rear, and the emergency braking sounds of surrounding vehicles.

[0172] S213. Divide the training set, test set, and validation set.

[0173] The driving sound data will be divided into equal-length sound wavelength data according to the sliding window method. The sliding window is w , and the sliding step is s ; The calculation formula for the number of samples in each sound data is shown in (1):

[0174] (1)

[0175] In the formula, is the length of the i-th sound data sample;

[0176] N The calculation formula (2) for the total number of samples of the driving sound signal data is:

[0177] (2)

[0178] Finally, the samples are divided into a training set, a test set, and a validation set according to a ratio. For example: the training set is 70%, which is used for model training, the test set is 20%, which is used for parameter tuning and model selection, and the test set is 10%, which is used for the final evaluation of the model.

[0179] S22. Build a driving sound recognition model that fuses time-frequency domain features;

[0180] S221. Temporal feature representation learning based on the attention mechanism;

[0181] Extract time series features from the time domain waveform signal; Sound waveform signal data with timestamps X ={ x 1, x 2,…, x T}, where x t is the waveform signal at time t , T is the time step; The original sound waveform signal , and the waveform signal after position encoding is expressed by formula (3):

[0182] (3)

[0183] Extract features through the encoder. The encoder consists of multiple layers of multi-head self-attention mechanisms and feed-forward neural networks, and outputs the feature representation result H t , ;

[0184] (4)

[0185] S222. Frequency domain feature representation learning based on CNN;

[0186] Perform a short-time Fourier transform on X to obtain a spectrogram , where F is the frequency dimension, is the number of time frames; obtain an intermediate feature representation result through multi-layer 2D convolution and pooling operations P , and the calculation formula (5) is:

[0187] (5)

[0188] Finally, obtain a frequency-domain feature representation result through dimensionality transformation and flattening operations H f , ;

[0189] S223. Fuse features based on the cross-attention mechanism;

[0190] Perform fusion on the time-domain feature representation result H t obtained by S221 through the cross-attention mechanism, H f and the frequency-domain feature representation result H fused of S222 to obtain a time-frequency domain fusion representation result

[0191] (6)

[0192] The calculation formula (7) of CrossAttention is:

[0193] (7)

[0194] Generate a corresponding W Q vector through a learnable weight matrix H t by performing a linear transformation on Q , and generate a corresponding W K vector through weight matrices W V and H f by performing a linear transformation on K , V ;

[0195] S224. Generate a driving signal recognition text;

[0196] Generate a word probability distribution through a linear projection and Softmax by means of a transformer decoder; the decoder consists ofl layers, including self-attention, cross-attention, and feed-forward networks; for the l decoder layer, the input is the output of the previous layer, and the initial input is the embedding of the target sequence Y = y 1, y 2,…, y (t-1) , plus the positional encoding;

[0197] For the processing of the self-attention sublayer, the decoding result Zs is obtained through residual connection and layer normalization. The calculation formula (8) of Zs is:

[0198] (8)

[0199] where , , , W Qs , W Ks and W Vs are learnable parameters;

[0200] For the processing of the cross-attention sublayer, the decoding result Zc is obtained. The calculation formula (9) of Zc is:

[0201] (9)

[0202] where , , , W Qc , W Kc and W Vc are learnable parameters;

[0203] Then, through the feed-forward network sublayer, the calculation formula (10) is:

[0204] (10)

[0205] After residual connection and layer normalization, the representation result of the l layer is obtained. The calculation formula (11) is:

[0206] (11)

[0207] The word probability distribution is generated through linear projection and Softmax. The calculation formula (12) is:

[0208] (12)

[0209] Finally, realize the text output of the voice signal.

[0210] S23. Perform performance test and evaluation on the driving sound recognition model;

[0211] S231. Perform offline test and evaluation on the model;

[0212] Offline test and evaluation of the model. In this link, the Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) are used as the evaluation indicators of the driving sound recognition model to evaluate the difference between the predicted value and the true value of the voice signal recognition model. The calculation formulas (13), (14), and (15) of MAE, RMSE, and MAPE are as follows:

[0213] (13)

[0214] (14)

[0215] (15)

[0216] In the formula, and are the true voice signal text and the predicted voice signal text respectively;

[0217] S232. Perform in-loop test and evaluation on the model;

[0218] Deploy the trained model to the vehicle hardware; ensure that the model can process voice data in real time when running on the hardware and feedback the results in a timely manner; and verify the real-time recognition ability of the model for key sounds (such as ambulance sirens, collision sounds, abnormal tire noise, emergency braking sounds) in the real driving environment.

[0219] S3. Build a vehicle head-up display system for hearing-impaired drivers;

[0220] S31. Define the architecture of the vehicle head-up display system;

[0221] S311. Design system indicators and parameters;

[0222] Define the head-up display system indicators, including optical performance parameters such as the resolution and contrast of the displayed information, horizontal and vertical field of view angles, brightness, and chromaticity uniformity. It is necessary to meet the test under temperature and humidity conditions to ensure the reliability and safety in various environments;

[0223] S312. Compose the head-up display system;

[0224] The head-up display system includes a MEMS scanning mirror, a laser, a laser diode driver, and a projection screen; the driver enables flexible adjustment of the shape and brightness of the laser beam, and the scanner controls the light source provided by the laser to achieve high-resolution image projection.

[0225] S313. Generate MEMS laser projection imaging;

[0226] Combining microelectromechanical systems and laser scanning technology, using red, green, and blue (RGB) three primary color lasers as the light source, the laser is collimated, focused, and modulated by optical elements (such as lenses, mirrors, etc.). After passing through the MEMS micromirror, the laser beam is scanned according to the pixel information of the image data, and finally a complete image is formed on the screen;

[0227] S32. Obtain vehicle head-up display information;

[0228] S321. Convert the sound signal into text information;

[0229] Deploy the driving sound recognition model that combines time-frequency domain features constructed in S22 on the vehicle hardware. The driving sound signal data obtained by the on-vehicle sound sensor is input into the vehicle controller VCU through the CAN bus, and after processing, the corresponding text information is output;

[0230] S322. Obtain vehicle driving state information;

[0231] Obtain acceleration and angular velocity through an inertial navigator, and perform integral processing to obtain speed and position information; monitor the voltage, current, and temperature of the battery through the battery management system, and calculate the battery capacity and driving range driving state information;

[0232] S323. Generate vehicle collision warning information;

[0233] The original data of the target object obtained by lidar, millimeter-wave radar, ultrasonic radar, and camera is processed and fused, combined with the speed and acceleration of the vehicle itself, calculate the relative motion relationship with the target object, and predict the collision time to generate vehicle collision warning information;

[0234] S324. Integrate vehicle head-up display information;

[0235] Integrate the sound signal text information, driving state information, and vehicle collision warning information output by S321, S322, and S323; the VCU sends the information to the head-up display system controller according to a fixed data format and message format, the head-up display system controller receives the signal, converts the received signal into a visual graph, and projects it on the screen through the optical projection system.

[0236] S33. Design a vehicle head-up display strategy for hearing-impaired drivers.

[0237] S331. Design the head-up display interface;

[0238] Project the display information to different positions on the display screen according to different priorities; take the driving sound signal as the first priority and project it to the center of the screen where the driver is looking. The display information of the second priority is shown below the center of the screen, and the display information of the fourth and third priorities are shown on the left and right sides respectively.

[0239] S332. Determine the priority of the vehicle head-up display information;

[0240] S4. The driver assists in driving according to the display information of the vehicle head-up display system.

[0241] Put the sound signal information that hearing-impaired drivers cannot obtain in the first priority to help drivers understand the driving environment around them. The display information of the second priority includes collision warning information, vehicle battery power shortage prompt information, lane departure information, and fault information. The third priority displays navigation information, including lane information, lane line information, traffic light information, and traffic sign information. The fourth priority displays external environment interaction information, including phone call display information, weather information, and in-vehicle temperature information.

[0242] Embodiment 2:

[0243] This embodiment provides a vehicle head-up display system for hearing-impaired drivers, which is used to implement the vehicle head-up display method for hearing-impaired drivers described in Embodiment 1, and includes:

[0244] An acquisition module, which is used to acquire sound signals for complex driving environments;

[0245] A first construction module, which is used to construct a voice recognition model for hearing-impaired drivers;

[0246] A second construction module, which is used to construct a vehicle head-up display system for hearing-impaired drivers. [[ID=Q3]]

[0247] In summary, the present invention enables hearing-impaired drivers to better perceive the surrounding driving environment, improves driving safety, and solves the problem that hearing-impaired drivers cannot perceive the sound information of the surrounding driving environment.

[0248] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirits of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle head-up display method for hearing-impaired drivers, Characterized in that it includes the following steps: S1. Obtain sound signals for complex driving environments, and the specific method is as follows: S11. Obtain sensor-based sound signals; S12. Obtain driving sound signals of web crawlers; S13. Obtain sound signals based on generative adversarial networks, specifically as follows: S131. Preprocess the sound signals; Use a noise reduction algorithm to remove environmental noise in the audio files of S11 and S12; divide the driving scenario sound signals into short-time frames, perform Fourier transform on each frame of the signal to obtain the spectrum; subtract the noise spectrum from the spectrum of the noisy signal, and finally convert the processed spectrum back to the time-domain signal; then perform standardization or normalization processing on the noise-reduced sound data, and scale the waveform data to between [-1, 1]; S132. Construct a sound generation model; The sound generation model includes a generator and a discriminator. Use one-dimensional transposed convolution to construct the generator and one-dimensional convolution to construct the discriminator; the input of the generator is a random noise vector, and the output is the generated sound waveform data; the generator includes multiple fully connected layers or convolutional layers; first, map the random noise data through the fully connected layer to a middle dimension; use the deconvolution layer to gradually upsample the data to the dimension of the target waveform data; the output layer outputs the generated sound waveform data, and the dimension is the same as the real data; the input of the discriminator is real driving sound signal data or fake data generated by the generator, and the output is a scalar representing the probability that the input data is real data; use the binary cross-entropy loss function to alternately train the generator and the discriminator, so that the discriminator can correctly distinguish real data and fake data, and the fake data generated by the generator can deceive the discriminator into thinking that the fake data is real, and finally generate data identical to the real driving sound waveform data; S133. Conduct reliability analysis on the sound signal data generated in S132; Post-process the generated sound signals; use objective indicators such as signal-to-noise ratio and logarithmic spectrum distance to evaluate the quality of the generated signals. The higher the signal-to-noise ratio, the better the signal quality, and the smaller the logarithmic spectrum distance, the closer the generated signal is to the target signal; and evaluate the quality of the generated signals through listening tests to ensure that the timbre, rhythm, and dynamic range of the driving sound signals conform to the auditory characteristics of the real scene; S134. Integrate the sound signal data for complex driving scenarios; Integrate the reliability driving sound data L generated by S11, S12, and S132, L = (L1, L2,..., L i ,..., L N ), a total of N driving sound signal data, where each sound waveform signal data L i has a different data length; S2. Construct a sound recognition model for hearing-impaired drivers, and the specific method is as follows: S21. Construct a driving scenario sound signal dataset; S22. Construct a driving sound recognition model that fuses time-frequency domain features; S23. Conduct performance test and evaluation on the driving sound recognition model; S3. Construct a vehicle head-up display system for hearing-impaired drivers, and the specific method is as follows: S31. Define the vehicle head-up display system architecture; S32. Obtain vehicle head-up display information, specifically as follows: S321. Convert the sound signal into text information; Deploy the driving sound recognition model that fuses time-frequency domain features constructed in S22 on vehicle hardware. The driving sound signal data obtained through in-vehicle sound sensors is input into the vehicle control unit (VCU) via the CAN bus, and after processing, the corresponding text information is output. S322. Obtain the vehicle driving status information. Obtain the acceleration and angular velocity through an inertial navigator, and perform integration processing to obtain the speed and position information; monitor the voltage, current, and temperature of the battery through the battery management system to obtain the battery capacity and driving range. S323. Generate vehicle collision warning information. The original data of the target object obtained by lidar, millimeter-wave radar, ultrasonic radar, and camera, after data processing and fusion, combined with the speed and acceleration of the vehicle itself, calculate the relative distance d and relative speed v from the target object rel , and predict the time to collision ttc, ttc = d / v rel , and generate a vehicle collision warning message; S324. Integrate the vehicle head-up display information. Integrate the sound signal text information, driving status information, and vehicle collision warning information output by S321, S322, and S323. The VCU sends the information to the head-up display system controller according to a fixed data format and message format. The head-up display system controller receives the signal, converts the received signal into a visual graph, and projects it onto the screen through an optical projection system. S33. Design a vehicle head-up display strategy for hearing-impaired drivers, specifically as follows: S331. Design the head-up display interface. Project the display information to different positions on the display screen according to different priorities; take the driving sound signal as the first priority and project it to the center of the screen where the driver is looking. The display information of the second priority is shown below the center of the screen, and the display information of the fourth and third priorities is shown on the left and right sides respectively. S332. Determine the priority of the vehicle head-up display information. Put the sound signal information that hearing-impaired drivers cannot obtain in the first priority to help the driver understand the driving environment around; the display information of the second priority includes collision warning information, vehicle battery low warning information, lane departure information, and fault information; the third priority displays navigation information, including lane information, lane line information, traffic light information, and traffic sign information; the fourth priority displays external environment interaction information, including phone call display information, weather information, and in-vehicle temperature information.

2. A vehicle head-up display method for hearing-impaired drivers according to claim 1, Its feature lies in that the specific method of S11 is as follows: S111. Arrange vehicle sound sensors. Install one sound sensor on each side of the vehicle head for collecting the sound signals in the front and on the sides; install one sound sensor on each side of the vehicle rear for obtaining the vehicle horn sounds from the rear and the side-rear. S112. Define the acquisition method. According to the Nyquist sampling theorem, set the sampling rate to twice the highest frequency of the signal, and all frequency sound signals have the same sensitivity; adopt multiple channels and ensure that the sound sensors have the same sensitivity to sound signals in different directions. S113. Perform digital processing on the sound signals. Sample the analog signal at a fixed time interval to discretize the continuous signal; convert the amplitude value of each sampling point into a digital value with a finite precision; convert the quantized digital value into a binary code, and the coding method used is PCM. S114. Compress and store the data of the sound signals. Adopt the method of lossless compression and local storage.

3. A vehicle head-up display method for hearing-impaired drivers according to claim 1, characterized in that, The specific method of S12 is as follows: S121. Grab the sound data based on keywords. Using web crawler technology, capture driving sound signal data from the Internet based on keywords; First, determine the target website and data source, analyze the HTML structure of the target web page, and locate the position where the target data is located; Extract the links, titles, descriptions, and tags of the target data from the search result page; S122. Download sound data based on multi-threading; Implement multi-threading using the method of thread pool; Use the sound signal web parsing module and download module to maintain a thread pool respectively; According to the remaining capacity of the thread pool, allocate a new thread. When the web parsing task or download task is completed, destroy the thread, and at this time the current remaining capacity is incremented by 1; When it is detected that the remaining capacity of the thread pool is 0, wait for the parsing task or download task to complete and then release the resources, and then allocate resources to the new thread; S123. Store and classify the sound signals; Store the metadata information of the audio file in a database or file; Convert the downloaded audio files into a unified format; Classify the audio files, and label each segment according to the timestamp; Finally, upload the data to the cloud storage service.

4. A vehicle head-up display method for hearing-impaired drivers according to claim 1, characterized in that The specific method of S21 is as follows: S211. Extract data features; Extract features from the unequal-length sound waveform data integrated in S134, including time-domain feature extraction and frequency-domain feature extraction; The time-domain features include mean, variance, and peak value; Use the fast Fourier transform to convert the time-domain sound signal into a frequency-domain sound signal, expressed as a two-dimensional array of frequency and time, calculate the power spectrum from the complex matrix of the Fourier transform, and perform matrix multiplication on the power spectrum and the Mel filter bank to obtain the Mel spectrogram, where the horizontal axis is time and the vertical axis is the Mel frequency; S212. Label the data; Label the collected driving sound data; Classify and organize the driving sound data according to the scenario and sound category, and assign one or more category labels to the sound in each time period; S213. Divide the training set, test set, and validation set; Divide the driving sound data into equal-length sound wavelength data according to the sliding window method, where the sliding window is w and the sliding step is s; The calculation formula for the number of samples in each sound data sample is shown in (1): where L i is the length of the i-th sound data sample; The calculation formula (2) for the total number of N driving sound signal data samples is: Finally, divide the samples into the training set, test set, and validation set according to the ratio.

5. A vehicle head-up display method for hearing-impaired drivers according to claim 1, characterized in that, The specific method of S23 is as follows: S231. Conduct offline test and evaluation on the model; Use the mean absolute error, root mean square error, and mean absolute percentage error as the evaluation indicators of the driving sound recognition model to evaluate the difference between the predicted value and the true value of the sound signal recognition model; The calculation formulas (13), (14), and (15) of MAE, RMSE, and MAPE are as follows: where y i and y i ′ are the true voice signal text and the predicted voice signal text, respectively; S232. Conduct in-loop test and evaluation on the model; Deploy the trained model to the in-vehicle hardware; Ensure that the model can process sound data in real time when running on the hardware and feedback the results in a timely manner; And verify the real-time recognition ability of the model for key sounds in the real driving environment.

6. The vehicle head-up display method for hearing-impaired drivers according to claim 1, characterized in that The specific method of S31 is as follows: S311. Design system indicators and parameters; Define the head-up display system metrics, including the resolution and contrast of the displayed information, the horizontal and vertical field of view angles, the brightness, and the chromaticity uniformity; S312. Compose the head-up display system; The head-up display system includes a MEMS scanning mirror, a laser, a laser diode driver, and a projection screen; S313. Generate MEMS laser projection imaging; Combining microelectromechanical systems and laser scanning technology, using red, green, and blue primary color lasers as light sources, collimating, focusing, and modulating the lasers through optical elements; after the laser beam passes through the MEMS micromirror, it is scanned according to the pixel information of the image data, and finally a complete image is formed on the screen.

7. A vehicle head-up display system for hearing-impaired drivers, which is used to implement the vehicle head-up display method for hearing-impaired drivers according to any one of claims 1-6, characterized in that Including: An acquisition module for acquiring sound signals for complex driving environments; A first construction module for constructing a voice recognition model for hearing-impaired drivers; A second construction module for constructing a vehicle head-up display system for hearing-impaired drivers.

Images