Vehicle auxiliary driving method and device, electronic equipment, storage medium and vehicle

Abstract

The invention discloses a vehicle auxiliary driving method and device, electronic equipment, a storage medium and a vehicle, and relates to the technical field of auxiliary driving. According to the vehicle aided driving method provided by the invention, the semantic tag is obtained by fusing the acoustic feature data and the vehicle dynamic data for comprehensive semantic analysis, the semantic fuzziness problem caused by a single sound feature is avoided, and the semantic recognition accuracy in a complex scene is improved. According to the semantic label and the vehicle dynamic data, multi-sense collaborative feedback is performed through the photoelectric device and the vibration device, and the information receiving efficiency of the deaf-mute driver is improved. And the multi-sense collaborative feedback can shorten the understanding time of the deaf-mute driver on complex instructions.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and in particular to a vehicle driver assistance method, device, electronic device, storage medium, and vehicle. Background Technology

[0002] With the development of driver assistance systems (ADAS), existing technologies often interact with or provide reminders to drivers via voice. However, deaf and mute individuals, due to their hearing and speech impairments, cannot interact with ADAS devices through voice. Furthermore, their hearing impairments prevent them from recognizing audible cues from surrounding vehicles, potentially causing them to miss crucial information and leading to accidents.

[0003] Assisted driving technologies for the deaf and mute generally adopt a mapping rule of "voice features - simple commands", such as the number of horn blasts corresponding to text prompts. This can only convey basic information such as "present" or "number of times", and cannot cover the multi-dimensional data required in driving scenarios. Moreover, the mapping rules are mostly general presets and do not take into account the differentiated needs of different driving scenarios. Once interference or misrecognition occurs, it is very easy for users to make incorrect operations, thus limiting the practicality. Summary of the Invention

[0004] The purpose of this application is to provide a vehicle-assisted driving method, device, electronic device, storage medium, and vehicle that can avoid the semantic ambiguity caused by single sound features and improve the semantic recognition accuracy in complex scenarios. It can also perform multi-sensory collaborative feedback to improve the information reception efficiency of deaf and mute drivers. Multi-sensory collaborative feedback can shorten the time it takes for deaf and mute drivers to understand complex instructions.

[0005] This application provides the following solution:

[0006] According to one aspect of this application, a vehicle-assisted driving method is provided, applicable to an intelligent cockpit recommendation system, the method comprising:

[0007] Acoustic signal data around the vehicle is collected using an acoustic array, as well as vehicle dynamic data.

[0008] Extract acoustic feature data from the acoustic signal data;

[0009] The acoustic feature data and the vehicle dynamic data are input into a preset semantic reasoning model to obtain the semantic labels corresponding to the acoustic signal data;

[0010] Based on the semantic tags and the vehicle dynamic data, the photoelectric device and the vibration device are controlled to provide visual and tactile feedback, respectively.

[0011] In an optional embodiment, the acquisition of acoustic signal data around the vehicle via an acoustic array includes:

[0012] Acoustic signal data around the vehicle is collected by microphones positioned at the front, rear, left, and right of the vehicle.

[0013] In an optional embodiment, acquiring vehicle dynamic data includes:

[0014] The relative speed, relative distance, and azimuth data of the vehicles are obtained through radar.

[0015] The system uses cameras to capture lane line data, traffic sign data, and traffic light status data of the vehicle's surroundings.

[0016] Vehicle speed data, steering angle data, and braking signal data are obtained through the vehicle bus.

[0017] In an optional embodiment, extracting acoustic feature data from the acoustic signal data includes:

[0018] The acoustic signal data is subjected to noise suppression processing to obtain the sound source signal;

[0019] The speaker sound source signal is located using a delay-sum beamforming algorithm to obtain the sound source azimuth angle and sound source distance.

[0020] Extract the sound source duration, sound source pulse count, and sound source center frequency from the sound source signal;

[0021] The acoustic feature data includes: sound source azimuth angle, sound source distance, sound source duration, sound source pulse count, and sound source center frequency.

[0022] In an optional embodiment, inputting the acoustic feature data and the vehicle dynamic data into a preset semantic reasoning model includes:

[0023] The driving scenario of the vehicle is determined based on the vehicle dynamic data;

[0024] The coefficients corresponding to the driving scenario are determined as the weights of the acoustic feature data;

[0025] Acoustic fusion data is generated based on the weights and the acoustic feature data;

[0026] The acoustic fusion data and the vehicle dynamic data are input into a preset semantic reasoning model.

[0027] In one optional embodiment, the photoelectric device includes: a left door ambient light, a right door ambient light, a center console ambient light, and an interior rearview mirror side light;

[0028] The vibration device includes a motor on the right side of the steering wheel and a seat vibration motor.

[0029] According to two aspects of this application, a vehicle driver assistance device is provided, the device comprising:

[0030] The acoustic acquisition unit collects acoustic signal data around the vehicle through microphones positioned at the front, rear, left, and right of the vehicle, and amplifies and filters the acoustic signal data through a signal conditioning circuit.

[0031] The dynamic data acquisition unit acquires vehicle relative speed, relative distance, and azimuth data via radar; acquires lane line data, traffic sign data, and traffic light status data of the vehicle's environment via camera; and acquires vehicle speed data, steering angle data, and braking signal data via the vehicle bus.

[0032] The visual feedback unit, used for providing visual feedback, is connected to the vehicle's lighting drive module via the SPI bus and includes: ambient lights for the left and right doors, ambient lights for the center console, and ambient lights next to the interior rearview mirror.

[0033] The haptic feedback unit, used for vibration feedback, is connected to the vehicle's vibration drive module via an I²C bus and includes a four-way linear motor installed in the steering wheel and an eight-way vibration motor installed in the seat.

[0034] The main control unit is used to execute the steps of the vehicle assisted driving method.

[0035] According to three aspects of this application, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0036] The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the vehicle's driver assistance method.

[0037] According to four aspects of this application, a computer-readable storage medium is provided, comprising: storing a computer program executable by an electronic device, wherein when the computer program is run on the electronic device, it causes the electronic device to perform the steps of a vehicle-assisted driving method.

[0038] According to five aspects of this application, a vehicle is provided, comprising:

[0039] Electronic devices used to implement vehicle driver assistance methods;

[0040] The processor runs programs, and when the programs run, they execute the steps of vehicle-assisted driving methods based on data output from electronic devices.

[0041] Storage medium for storing programs that, when run, execute steps of a vehicle-assisted driving method based on data output from an electronic device.

[0042] The above solution achieves the following beneficial technical effects:

[0043] The vehicle-assisted driving method provided in this application obtains semantic tags by fusing acoustic feature data and vehicle dynamic data for comprehensive semantic analysis. This avoids the semantic ambiguity caused by single sound features and improves the accuracy of semantic recognition in complex scenarios. Based on the semantic tags and vehicle dynamic data, multi-sensory collaborative feedback is achieved through photoelectric devices and vibration devices, improving the information reception efficiency of deaf and mute drivers. Multi-sensory collaborative feedback can shorten the time for deaf and mute drivers to understand complex instructions. Attached Figure Description

[0044] Figure 1 This is a schematic flowchart of the vehicle assisted driving method provided in the embodiments of this application.

[0045] Figure 2 This is a schematic diagram of the vehicle driver assistance device provided in the embodiments of this application.

[0046] Figure 3 This is a block diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The assisted driving technology for the deaf and mute adopts the mapping rule of "voice features - simple commands". The core pain points in the interaction design are mainly the following problems.

[0049] 1. The information dimension is limited, only conveying basic information such as "presence" and "frequency," failing to cover the multidimensional data required in driving scenarios. For example, whether a vehicle approaching from behind is in the left or right lane, or the direction of the horn's origin, cannot be indicated by a simple command. Furthermore, the existing mapping cannot distinguish between the urgency of a normal horn signal and an emergency evasive horn. The "type" information of different sounds, such as ambulance sirens and construction warning sounds, cannot be conveyed to the user through a single command.

[0050] 2. The mapping rules are mostly general presets and do not take into account the differentiated needs of different driving scenarios, which limits their practicality.

[0051] For example, in congested areas, frequent short honks can generate a lot of redundant instructions in the "number mapping," interfering with the user's judgment; while on highways, the key information of a single long honk may be ignored due to rule simplification.

[0052] 3. Simple mapping lacks a verification mechanism, and once interference or misidentification occurs, it is very easy for users to make incorrect operations.

[0053] External environmental noise (such as horns from roadside shops) may be mistaken for valid honking, triggering incorrect commands. Most solutions exhibit a high rate of misidentification in noisy environments such as rainy days or tunnels. After receiving a command, the user cannot confirm whether they have accurately understood it through reverse interaction; if the command is transmitted incorrectly, it cannot be corrected in time. Most solutions rely solely on visual cues (such as text and lights); if the user's gaze temporarily leaves the cues area, they will completely miss crucial information, lacking multi-channel redundancy protection.

[0054] To address the problems existing in the prior art, this application provides a vehicle-assisted driving method, device, electronic device, storage medium, and vehicle, which improves the accuracy of semantic recognition in complex scenarios. It also enables multi-sensory collaborative feedback, improving the information reception efficiency for deaf and mute drivers. Multi-sensory collaborative feedback can shorten the time it takes for deaf and mute drivers to understand complex instructions.

[0055] Firstly, embodiments of this application provide a vehicle assisted driving method, see [link to relevant documentation]. Figure 1 The vehicle's driver assistance methods include:

[0056] S101: Acquires acoustic signal data around the vehicle and obtains vehicle dynamic data through an acoustic array.

[0057] In this step, acoustic information about the vehicle's surrounding environment and its own motion state information are acquired to provide basic data for subsequent analysis.

[0058] Specifically, acoustic signal data around the vehicle are collected using microphones positioned in front, behind, to the left, and to the right of the vehicle. The microphones also collect non-visual information such as horn sounds, collision sounds, and pedestrian shouts.

[0059] Specifically, radar acquires data on the vehicle's relative speed, relative distance, and azimuth; cameras acquire data on lane markings, traffic signs, and traffic light status in the vehicle's environment; and the vehicle's speed, steering angle, and braking signal data are acquired via the vehicle bus.

[0060] It should be noted that by adding vehicle dynamic data (such as vehicle speed, steering angle, and braking status), subsequent analysis can be combined with the vehicle's own status, avoiding judgments that are divorced from actual driving scenarios.

[0061] S102: Extract acoustic feature data from the acoustic signal data.

[0062] In this step, valuable feature information is filtered from the raw acoustic signal data, and useless noise is removed to reduce the computational load of subsequent models. In this embodiment, the feature information refers to the horn sound signal; the pure horn sound signal is extracted from the acoustic signal data.

[0063] Specifically, noise suppression processing is performed on acoustic signal data to obtain the sound source signal. An improved Wave-U-Net deep learning model is then used (the input is the time-domain signal from four microphones, and the output is the noise-reduced, clean horn sound signal). The Wave-U-Net deep learning model is trained on 100,000 noisy samples (scenes such as rainy days, tunnels, and urban congestion), and it can still maintain a signal fidelity of over 85% even at a signal-to-noise ratio of 10dB. The Wave-U-Net deep learning model improves the horn sound recognition rate in scenarios such as rainy days, tunnels, and urban congestion, and reduces the false recognition rate in extreme environment modes.

[0064] The speaker sound source signal is located using a delay-sum beamforming algorithm. The time difference of arrival (TDOA) of the signals from the four microphones is calculated. Combined with the vehicle coordinate system (with the driver's position as the origin), the azimuth angle and distance of the sound source signal are output. The azimuth angle ranges from -180° to +180°, and the distance ranges from 0 to 50m. The positioning error is ≤1m.

[0065] Extract the time-domain features (sound source duration, number of sound source pulses) and frequency-domain features (sound source center frequency, spectral energy distribution) from the sound source signal. For example, "one short press" is defined as a signal with a duration of 0.2-0.5s and a center frequency of 1kHz±200Hz.

[0066] It should be noted that the acoustic characteristic data includes: sound source azimuth angle, sound source distance, sound source duration, sound source pulse count, and sound source center frequency.

[0067] In this step, the original acoustic signal data contains a lot of interference (such as wind noise and engine noise). After extracting features (horn sound signal), it provides "high-quality input" to the semantic reasoning model, avoiding the model's judgment delay or error due to processing redundant data.

[0068] S103: Input the acoustic feature data and the vehicle dynamic data into a preset semantic reasoning model to obtain the semantic label corresponding to the acoustic signal data.

[0069] In this step, acoustic feature data and vehicle dynamic data are input into the semantic reasoning model to obtain semantic labels, thus transforming "feature data" into understandable "semantic labels" to clarify the actual event corresponding to the horn sound. That is, the semantic reasoning model analyzes the acoustic feature data and vehicle dynamic data and outputs specific semantic labels, such as "vehicle horn behind" or "vehicle horn to the left".

[0070] Before inputting acoustic feature data and vehicle dynamic data into the semantic reasoning model, the acoustic feature data and vehicle dynamic data need to be aligned using timestamps, and the synchronization error needs to be less than or equal to 10ms.

[0071] The semantic reasoning model in this step is pre-trained based on a labeled dataset, which includes acoustic feature data and vehicle dynamic data, as well as semantic labels that annotate the acoustic feature data and vehicle dynamic data.

[0072] The semantic reasoning model employs a lightweight Transformer model (8 million parameters, inference latency ≤ 300ms). It takes fused multimodal features (acoustic feature data and vehicle dynamic data) as input and outputs semantic labels (such as "left-hand overtaking request," "forward avoidance warning," etc., a total of 12 preset semantic categories) and confidence levels (0-100%). The semantic reasoning model is trained on a labeled dataset (containing over 50,000 real-world driving scenarios), achieving a test set accuracy greater than or equal to 92%.

[0073] Furthermore, before inputting the acoustic feature data into the semantic reasoning model, the vehicle's driving scenario can be determined based on the vehicle dynamic data; the coefficients corresponding to the driving scenario can be determined as the weights of the acoustic feature data; acoustic fusion data can be generated based on the weights and acoustic feature data; and the acoustic fusion data and vehicle dynamic data can be input into the preset semantic reasoning model.

[0074] A scene-semantic association model is constructed through a weighting mechanism to assign dynamic weights to acoustic feature data in different scenes. For example, in a highway scene (vehicle speed greater than or equal to 80 km / h): the weight of "danger warning" for "a long horn blast from behind (lasting greater than or equal to 2 seconds) plus a relative speed greater than or equal to 10 km / h" is 0.9; in an intersection scene (vehicle speed less than or equal to 30 km / h and traffic lights are detected): the weight of "reminder to start" for "a short horn blast from behind (0.2-0.5 seconds)" is 0.8.

[0075] It should be noted that when aligning acoustic feature data, vehicle dynamic data, and driving scenarios (labels such as "highway," "intersection," and "parking lot" identified by the camera) using timestamps, the synchronization error needs to be less than or equal to 10ms.

[0076] By integrating acoustic feature data, vehicle dynamic data, and driving scene data, this approach addresses the semantic ambiguity problem inherent in single-modal analysis. Compared to relying solely on sound features, it improves the accuracy and precision of semantic recognition in complex scenarios.

[0077] In this step, the sound from the loudspeaker can be translated into specific labels, providing a clear basis for subsequent control decisions.

[0078] S104: Control the photoelectric device and the vibration device to provide visual feedback and tactile feedback respectively according to the semantic tag and the vehicle dynamic data.

[0079] In this step, the photoelectric devices include: ambient light on the left door, ambient light on the right door, ambient light on the center console, and side light on the rearview mirror; the vibration devices include: a motor on the right side of the steering wheel and a seat vibration motor.

[0080] The semantic label and signal mapping rules in this step are as follows:

[0081] The semantic tags for reminder categories (such as "overtaking request", with a confidence level greater than 80%) correspond to the following mapping rules: yellow light (wavelength 580nm), low-frequency flashing (1Hz), low motor amplitude (0.5mm), and low-frequency vibration (50Hz).

[0082] The semantic tags for warning categories (such as "danger approaching", with a confidence level greater than 80%) correspond to the following mapping rules: red light (wavelength 620nm), high-frequency flashing (3Hz), high-amplitude motor vibration (2mm), and high-frequency vibration (200Hz).

[0083] The uncertain category (confidence level 60%-80%) in semantic tags corresponds to the following mapping rules: orange light (wavelength 600nm), slow flash (0.5Hz), and text pop-up (the central control screen displays "sound on the left, meaning to be confirmed").

[0084] Furthermore, based on the detected horn signal's location (front, rear, left, and right of the vehicle), a position-signal mapping rule is established, as follows:

[0085] The left-side sound source (azimuth angle -90° to -30°) corresponds to the following mapping rules: left door ambient light + left-side steering wheel motor + left-side seat vibration unit;

[0086] The right-side sound source (azimuth angle 30° to 90°) corresponds to the following mapping rules: right door ambient light + right-side steering wheel motor + right-side seat vibration unit;

[0087] The mapping rule for the front sound source (azimuth angle -30° to 30°) is: center console ambient lighting + steering wheel front motor + seat front vibration unit;

[0088] The rear sound source (azimuth angle 90° to 270°) corresponds to the following mapping rules: interior rearview mirror side light + steering wheel rear motor + seat rear vibration unit.

[0089] In this step, the main control unit that controls the photoelectric device and the vibration device generates PWM signals (light brightness adjustment) and pulse signals (vibration intensity / frequency adjustment) according to the mapping rules. The feedback unit is controlled by the drive module to execute the signal, and the response delay is required to be less than or equal to 100ms.

[0090] It is understandable that the location and signal mapping rules conform to human perception habits (such as vibration on the left side corresponding to a sound source on the left side), and the semantic labels and signal mapping rules conform to the physiological reaction patterns in emergency situations, thereby improving the average understanding time of instructions for deaf and mute drivers.

[0091] Furthermore, based on the above embodiments, safety redundancy control is added to ensure the reliability of the vehicle assisted driving method. Specifically, this includes:

[0092] Hardware fault detection: The main control unit reads the status registers of the microphone, radar and camera in real time through the I²C bus. When the sensor is detected to be offline (e.g., no signal output from the microphone for 1 second), it immediately triggers a "system abnormality" prompt (red solid light on the center console + long vibration of the entire seat area for 3 seconds) and records the fault code (stored in the MCU flash memory).

[0093] Semantic confidence filtering: When the confidence level of the inference result is less than 60%, the system does not output specific semantic signals, but only triggers an "unknown sound" prompt (white light flashing + slight steering wheel vibration once) to avoid misjudgment. That is, by reducing the risk of failure through safety redundancy control, driving safety is significantly improved.

[0094] As described above, the vehicle-assisted driving method provided in this application obtains semantic tags by fusing acoustic feature data and vehicle dynamic data for comprehensive semantic analysis. This avoids the semantic ambiguity caused by single sound features and improves the accuracy of semantic recognition in complex scenarios. Based on the semantic tags and vehicle dynamic data, multi-sensory collaborative feedback is achieved through photoelectric devices and vibration devices, improving the information reception efficiency for deaf and mute drivers. Multi-sensory collaborative feedback can shorten the time for deaf and mute drivers to understand complex instructions.

[0095] The technical solution provided in this application will be described in detail below through specific application examples, including:

[0096] Example 1: High-speed overtaking scenario.

[0097] 1. Triggering conditions:

[0098] A vehicle behind was detected honking its horn continuously for 2 seconds (center frequency 1.2kHz).

[0099] The relative speed of the vehicle behind was detected to be greater than 15 km / h, and the distance was less than 50 m;

[0100] It was detected that the vehicle was in a high-speed scene (vehicle speed greater than 100km / h, lane lines are dashed).

[0101] 2. Semantic reasoning:

[0102] The semantic reasoning model outputs the semantic label "rear emergency overtaking request" (confidence level of 92%).

[0103] Multisensory feedback:

[0104] Visual: The rearview mirror warning light flashes red at a high frequency (3Hz), and the left door ambient light emits a synchronized pulsed red light (brightness 80%).

[0105] Tactile sensation: High-amplitude vibration (2mm, 200Hz) from the linear motor behind the steering wheel, and synchronized high-frequency vibration from the vibration unit behind the seat back;

[0106] Spatial mapping: Vibration intensity diffuses from the back of the seat to both sides, simulating the spatial sensation of a vehicle approaching from behind.

[0107] 3. Safety redundancy:

[0108] The main control unit monitors the status of the millimeter-wave radar in real time. If the radar signal is lost, it automatically switches to camera visual ranging (error ±3m) and prompts "Radar abnormality, drive with caution" through vibration of the motor on the left side of the steering wheel.

[0109] Example 2: Starting at an intersection.

[0110] 1. Triggering conditions:

[0111] A short horn was detected from the vehicle behind (lasting 0.3 seconds, center frequency 1kHz).

[0112] The vehicle is currently stationary (speed is 0), and the camera detects that the traffic light ahead is green.

[0113] The millimeter-wave radar did not detect any obstacles ahead (distance greater than 10m).

[0114] 2. Semantic reasoning:

[0115] The semantic reasoning model outputs the semantic label "remind to start" (with a confidence level of 88%).

[0116] Multisensory feedback:

[0117] Visual: The ambient light on the center console flashes yellow at a low frequency (1Hz), and the ambient lights on the left and right doors emit a soft yellow light in sync;

[0118] Tactile feedback: Low-amplitude vibration (0.5mm, 50Hz) from the linear motor in front of the steering wheel, and synchronized low-frequency vibration from the vibration unit in front of the seat;

[0119] Interactive prompts: The central control screen displays the text "Starting from behind". Furthermore, gesture animations can be displayed through the speech synthesis module (which supports text-to-sign language animations for the deaf and mute).

[0120] 3. Safety redundancy:

[0121] If the camera misjudges the traffic light status (e.g., due to sunlight reflection causing recognition errors), and the millimeter-wave radar detects movement of a vehicle ahead, the system automatically triggers a "hazard warning" feedback (red light + high-frequency vibration) to override the original warning.

[0122] Example 3: Parking lot reversing scenario.

[0123] 1. Triggering conditions:

[0124] A horn was detected from a vehicle on the left rear (azimuth angle -120°, distance less than 5m).

[0125] The vehicle was detected to be in reverse gear (speed less than 5 km / h), and the reversing radar detected an obstacle on the left side at a distance of less than 2 meters.

[0126] The camera recognized that the current scene was a parking lot (parking lines, speed limit signs).

[0127] 2. Semantic reasoning:

[0128] The semantic inference model outputs the semantic label "Left-side vehicle warning and avoidance" (confidence level of 91%).

[0129] Multisensory feedback:

[0130] Visual: The ambient light on the left door flashes orange slowly (0.5Hz), and the warning light on the interior rearview mirror flashes orange in sync;

[0131] Tactile sensation: The linear motor on the left side of the steering wheel vibrates at a medium amplitude (1mm, 100Hz), and the vibration unit on the left side of the seat vibrates synchronously;

[0132] Spatial mapping: Vibrations diffuse from the left side of the seat towards the back, simulating the sense of direction when a vehicle is approaching from the left.

[0133] 3. Safety redundancy:

[0134] If the reversing radar gives a false alarm (such as interference from a metal obstacle), the system will visually identify the type of obstacle again through the camera. If it is a non-moving vehicle, the vibration intensity will be automatically reduced to 50%, and the message "Static obstacle on the left, keep your distance" will be displayed on the central control screen.

[0135] Secondly, embodiments of this application provide a vehicle driver assistance device, see [link to relevant documentation]. Figure 2 The vehicle's driver assistance system includes:

[0136] The acoustic acquisition unit collects acoustic signal data around the vehicle through microphones positioned at the front, rear, left, and right of the vehicle, and amplifies and filters the acoustic signal data through a signal conditioning circuit.

[0137] The microphones are four-channel MEMS (Micro-Electro-Mechanical System) microphones arranged in a linear array, positioned at the front, rear, left, and right of the vehicle. One microphone is installed on the front bumper, one on the inside of each of the left and right rearview mirrors, and one near the license plate frame at the rear. The microphone outputs are connected to a signal conditioning circuit via shielded cables for signal amplification and filtering.

[0138] The dynamic data acquisition unit acquires vehicle relative speed, relative distance, and azimuth data via radar; acquires lane line data, traffic sign data, and traffic light status data of the vehicle's environment via camera; and acquires vehicle speed data, steering angle data, and braking signal data via the vehicle bus.

[0139] The millimeter-wave radar is installed in the front grille and connected to the vehicle controller via the CAN (Controller Area Network) bus, outputting relative vehicle speed, distance, and azimuth data.

[0140] The camera is installed on the rearview mirror of the windshield and connected to the controller via the LVDS (Low-Voltage Differential Signaling) interface, outputting lane line, traffic sign and traffic light status data.

[0141] The vehicle-mounted CAN bus interface module collects the vehicle's own status (vehicle speed, steering angle, braking signal).

[0142] The visual feedback unit, used for providing visual feedback, is connected to the vehicle's lighting drive module via the SPI bus and includes ambient lights for the left and right doors, the center console, and the interior rearview mirror.

[0143] The ambient lights on the left and right doors, the center console, and the side lights of the interior rearview mirror are all connected to the vehicle's lighting driver module via the SPI (Serial Peripheral Interface) bus.

[0144] The haptic feedback unit, used for vibration feedback, is connected to the vehicle's vibration drive module via an I²C (Inter-Integrated Circuit) bus and includes a four-way linear motor installed in the steering wheel and an eight-way vibration motor installed in the seat.

[0145] The steering wheel has four built-in linear motors, corresponding to the front left, front right, rear left, and rear right areas respectively; the seats have eight built-in vibration motors, distributed at the four corners of the seat back and seat cushion, and connected to the vibration drive module via the I²C (Inter-Integrated Circuit) bus.

[0146] The main control unit is used to execute the steps of the vehicle assisted driving method.

[0147] The main control unit uses an automotive-grade MCU (Microcontroller Unit) as its core and connects to the AI ​​acceleration module via an Ethernet interface to achieve multi-module collaborative control. Power to each module or unit is provided by an onboard 12V to 5V power supply module to ensure stable power supply.

[0148] The vehicle assisted driving device provided in this application embodiment is used to implement the vehicle assisted driving method in the above embodiment. For details, please refer to the above vehicle assisted driving method, which will not be repeated here.

[0149] Thirdly, embodiments of this application provide an electronic device, such as... Figure 3 As shown, this application provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0150] The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the vehicle's driver assistance method.

[0151] Fourthly, this application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a vehicle-assisted driving method.

[0152] Fifthly, this application also provides a vehicle, comprising:

[0153] Electronic devices used to implement steps based on vehicle-assisted driving methods;

[0154] The processor runs programs, and when the programs run, they execute the steps of vehicle-assisted driving methods based on data output from electronic devices.

[0155] Storage medium for storing programs that, when run, execute steps of a vehicle-assisted driving method based on data output from an electronic device.

[0156] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0157] The electronic device includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory. The operating system can be any one or more computer operating systems that control the electronic device through processes, such as Linux, Unix, Android, iOS, or Windows. Furthermore, in this embodiment, the electronic device can be a smartphone, tablet, or other handheld device, or a desktop computer, portable computer, or other electronic device; there is no particular limitation in this embodiment.

[0158] In this embodiment, the executing entity for electronic device control can be an electronic device itself, or a functional module within an electronic device capable of calling and executing a program. The electronic device can obtain the firmware corresponding to the storage medium. This firmware is provided by the supplier, and different storage media may have the same or different firmware; this is not limited here. After obtaining the firmware corresponding to the storage medium, the electronic device can write this firmware into the storage medium; specifically, it burns the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented using existing technology, and will not be elaborated upon in this embodiment.

[0159] Electronic devices can also obtain reset commands corresponding to the storage media. The reset commands corresponding to the storage media are provided by the supplier. The reset commands corresponding to different storage media can be the same or different, and no restrictions are imposed here.

[0160] At this time, the storage medium of the electronic device is a storage medium on which the corresponding firmware has been written. The electronic device can respond to the reset command corresponding to the storage medium on which the corresponding firmware has been written, thereby resetting the storage medium on which the corresponding firmware has been written according to the reset command. The process of resetting the storage medium according to the reset command can be implemented by existing technology, and will not be described in detail in the embodiments of this application.

[0161] For ease of description, the above devices are described separately by function as various units and modules. Of course, in implementing this application, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0162] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0163] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of this application.

[0164] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A vehicle assisted driving method, characterized in that, The methods include: Acoustic signal data around the vehicle is collected using an acoustic array, as well as vehicle dynamic data. Extract acoustic feature data from the acoustic signal data; The acoustic feature data and the vehicle dynamic data are input into a preset semantic reasoning model to obtain the semantic labels corresponding to the acoustic signal data; Based on the semantic tags and the vehicle dynamic data, the photoelectric device and the vibration device are controlled to provide visual and tactile feedback, respectively.

2. The vehicle assisted driving method according to claim 1, characterized in that, The acquisition of acoustic signal data around the vehicle via an acoustic array includes: Acoustic signal data around the vehicle is collected by microphones positioned at the front, rear, left, and right of the vehicle.

3. The vehicle assisted driving method according to claim 1, characterized in that, The acquisition of vehicle dynamic data includes: The relative speed, relative distance, and azimuth data of the vehicles are obtained through radar. The system uses cameras to capture lane line data, traffic sign data, and traffic light status data of the vehicle's surroundings. Vehicle speed data, steering angle data, and braking signal data are obtained through the vehicle bus.

4. The vehicle assisted driving method according to claim 1, characterized in that, The extraction of acoustic feature data from the acoustic signal data includes: The acoustic signal data is subjected to noise suppression processing to obtain the sound source signal; The speaker sound source signal is located using a delay-sum beamforming algorithm to obtain the sound source azimuth angle and sound source distance. Extract the sound source duration, sound source pulse count, and sound source center frequency from the sound source signal; The acoustic feature data includes: sound source azimuth angle, sound source distance, sound source duration, sound source pulse count, and sound source center frequency.

5. The vehicle assisted driving method according to claim 1, characterized in that, The step of inputting the acoustic feature data and the vehicle dynamic data into a preset semantic reasoning model includes: The driving scenario of the vehicle is determined based on the vehicle dynamic data; The coefficients corresponding to the driving scenario are determined as the weights of the acoustic feature data; Acoustic fusion data is generated based on the weights and the acoustic feature data; The acoustic fusion data and the vehicle dynamic data are input into a preset semantic reasoning model.

6. The vehicle assisted driving method according to claim 1, characterized in that, The photoelectric device includes: ambient light for the left door, ambient light for the right door, ambient light for the center console, and side light for the interior rearview mirror; The vibration device includes a motor on the right side of the steering wheel and a seat vibration motor.

7. A vehicle driver assistance device, characterized in that, The device includes: The acoustic acquisition unit collects acoustic signal data around the vehicle through microphones positioned at the front, rear, left, and right of the vehicle, and amplifies and filters the acoustic signal data through a signal conditioning circuit. The dynamic data acquisition unit acquires vehicle relative speed, relative distance, and azimuth data via radar; acquires lane line data, traffic sign data, and traffic light status data of the vehicle's environment via camera; and acquires vehicle speed data, steering angle data, and braking signal data via the vehicle bus. The visual feedback unit, used for providing visual feedback, is connected to the vehicle's lighting drive module via the SPI bus and includes: ambient lights for the left and right doors, ambient lights for the center console, and ambient lights next to the interior rearview mirror. The haptic feedback unit, used for vibration feedback, is connected to the vehicle's vibration drive module via an I²C bus and includes a four-way linear motor installed in the steering wheel and an eight-way vibration motor installed in the seat. A main control unit for performing the steps of the vehicle assisted driving method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory stores a computer program that, when executed by a processor, causes the processor to perform the steps of the vehicle assisted driving method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The device stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the vehicle-assisted driving method as described in any one of claims 1 to 6.

10. A vehicle, characterized in that, include: An electronic device for implementing the steps of the vehicle assisted driving method as described in any one of claims 1 to 6; A processor that runs a program that, when the program is running, performs the steps of the vehicle-assisted driving method as described in any one of claims 1 to 6 from data output by the electronic device. A storage medium for storing a program that, when run, performs the steps of the vehicle-assisted driving method as described in any one of claims 1 to 6 on data output from an electronic device.

Images