Emergency vehicle processing method and device based on multiple modes, vehicle and medium

Abstract

The invention provides a multi-mode-based emergency vehicle processing method and device, a vehicle and a medium, and relates to the field of data / image processing, in particular to the fields of map navigation, automatic driving and intelligent transportation. According to the specific implementation scheme, the driving direction of the emergency vehicle and the relative position of the emergency vehicle and the vehicle are determined through collected audio data, and an avoidance strategy pushed to the emergency vehicle when emergency avoidance is needed is determined through collected image data of the emergency vehicle so that the emergency vehicle can be rapidly avoided when approaching the vehicle. By means of the method, the passing efficiency of emergency vehicles in the urban traffic environment is improved, and smoothness of a green'life channel 'is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent transportation in data / image processing, and in particular to a multimodal-based emergency vehicle processing method, device, vehicle and medium. Background Art

[0002] With the acceleration of urbanization, the number of vehicles in modern urban traffic environments is increasing. How to provide fast lanes for emergency vehicles in complex traffic environments is a technical challenge facing urban management and traffic planning.

[0003] Therefore, how to quickly avoid an emergency vehicle when it approaches your vehicle, improve the traffic efficiency of emergency vehicles in the urban traffic environment, and ensure the unobstructed green "life channel" is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure provides a method, device, vehicle and medium for handling emergency vehicles based on multi-modality.

[0005] According to a first aspect of the present disclosure, a multi-modal method for handling emergency vehicles is provided, comprising:

[0006] Based on the collected audio data, determine the direction of travel of the emergency vehicle and its relative position to the own vehicle;

[0007] Determining whether to collect image data of the emergency vehicle according to the relative position;

[0008] When it is determined that image data needs to be collected, image data obtained by collecting images of the location of the emergency vehicle is obtained;

[0009] Determining a lane relationship between a lane where the vehicle is located and a lane where the emergency vehicle is located based on the image data;

[0010] An avoidance strategy for the emergency vehicle is pushed according to the lane relationship, the driving direction and the relative position.

[0011] According to a second aspect of the present disclosure, a multi-modal processing device for emergency vehicles is provided, comprising:

[0012] A first processing unit, for determining a travel direction of the emergency vehicle and a relative position to the own vehicle based on the collected audio data;

[0013] A second processing unit, configured to determine whether to collect image data of the emergency vehicle according to the relative position;

[0014] A third processing unit is used to obtain image data obtained by collecting images of the position of the emergency vehicle when it is determined that image data needs to be collected;

[0015] a fourth processing unit, configured to determine a lane relationship between a lane where the vehicle is located and a lane where the emergency vehicle is located according to the image data;

[0016] The fifth processing unit is used to push an avoidance strategy for the emergency vehicle according to the lane relationship, the driving direction and the relative position.

[0017] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0021] According to a fourth aspect of the present disclosure, there is provided a vehicle, comprising: a vehicle body, and a plurality of audio collection devices respectively arranged at different positions of the vehicle body;

[0022] A processing device is arranged in the vehicle body, and is used to execute the multi-modal-based emergency vehicle processing method described in the first aspect; the processing device includes an audio processing unit and a vehicle control unit.

[0023] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0024] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 It is a scene diagram that can implement the multi-modal-based emergency vehicle handling method of the embodiment of the present disclosure;

[0028] Figure 2 is a flowchart of a multi-modal processing method for emergency vehicles provided in the first embodiment of the present disclosure;

[0029] Figure 3 is a flowchart of a multi-modal processing method for emergency vehicles provided in the second embodiment of the present disclosure;

[0030] Figure 4 is a flowchart of a method for handling emergency vehicles based on multimodality provided in the third embodiment of the present disclosure;

[0031] Figure 5 is a flowchart of a method for handling emergency vehicles based on multimodality provided in the fourth embodiment of the present disclosure;

[0032] Figure 6 is a flowchart of a multi-modal processing method for emergency vehicles provided in the fifth embodiment of the present disclosure;

[0033] Figure 7 is a structural block diagram of a multi-modal processing device for emergency vehicles provided in an embodiment of the present disclosure;

[0034] Figure 8 The block diagram is a block diagram of an electronic device for implementing a multi-modal method for handling emergency vehicles according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] Before introducing the embodiments of the present disclosure, the application background of the embodiments of the present disclosure is first explained:

[0037] With the acceleration of urbanization, road traffic volume has increased dramatically. Traffic congestion and frequent traffic accidents have become severe challenges facing urban transportation. The rapid passage of emergency vehicles is of great significance to protecting people's lives and property. However, ordinary drivers often fail to make timely avoidance due to their lack of clear judgment of the distance, direction and speed of emergency vehicles, which seriously affects the efficiency of emergency rescue.

[0038] The existing method of avoiding emergency vehicles is mainly to remind the vehicle in front to avoid by the horn or alarm of the emergency vehicle. On the one hand, due to the good sealing of the vehicle itself or the driver's concentration on driving or using Bluetooth devices to make calls, the driver cannot find the emergency vehicle in time and avoid it. On the other hand, although some in-vehicle navigation systems have the function of providing simple road condition notification, there is still a lack of effective solutions for accurate identification of emergency vehicles and real-time avoidance assistance in dynamic environments. In short, the existing method of avoiding emergency vehicles is difficult to provide sufficient passage space for emergency vehicles, resulting in the obstruction of emergency rescue tasks and reduced rescue efficiency.

[0039] To sum up, how to quickly avoid an emergency vehicle when it approaches your vehicle, improve the traffic efficiency of the emergency vehicle, and ensure the timeliness of rescue are technical problems that need to be solved urgently.

[0040] Based on the above technical problems, the inventors found that the above technical problems can be solved by combining the on-board audio acquisition device and the image acquisition device to obtain the position of the emergency vehicle in real time, analyzing the current lane information, generating an intelligent avoidance strategy, and guiding the vehicle to quickly make avoidance when the emergency vehicle approaches. This multi-sensor fusion avoidance strategy not only improves the accuracy of emergency vehicle identification, but also significantly improves the timeliness and effectiveness of avoidance, provides a smoother passage path for emergency vehicles, and ensures the efficient execution of rescue missions.

[0041] Based on this, the present disclosure provides a multimodal processing method, device, vehicle and medium for emergency vehicles, which are applied to map navigation, automatic driving and intelligent transportation in data / image processing, so as to achieve the technical effect of improving the traffic efficiency of emergency vehicles in urban traffic environment and ensuring the unobstructed green "life channel".

[0042] Figure 1It is a scene diagram that can implement the multimodal emergency vehicle handling method of the embodiment of the present disclosure, and the scene at least includes an ordinary vehicle (self-vehicle) 101, an emergency vehicle 102, and multiple audio collection devices 103 and image collection devices 104 mounted on the ordinary vehicle 101. Specifically, the audio collection device 103 collects audio data of the driving environment of the ordinary vehicle 101 in real time, determines whether there is an emergency vehicle 102 in the driving environment of the ordinary vehicle 101 based on the audio data, and determines the driving direction of the emergency vehicle 102 and the relative position of the self-vehicle 101, and determines whether it is necessary to start the image collection device 103 to obtain image data of the location of the emergency vehicle 102 according to the relative position. Finally, according to the image data, driving direction and relative position, the avoidance strategy for the emergency vehicle 102 is determined to provide a smooth passage path for the emergency vehicle 102.

[0043] The audio collection device 103 can be a microphone, a sound pressure sensor, an acoustic wave sensor, or other electronic device that can capture and process various sound signals in the environment; the image acquisition device 104 can be a camera, an infrared camera, a stereo vision camera, or other electronic device that can capture and process visual information in the environment; the emergency vehicle 102 can be an ambulance, a fire truck, a police car, an engineering rescue vehicle, or other vehicles that have the right of way when performing emergency rescue missions.

[0044] It should be noted that the physical devices involved in the above descriptions are exemplary in the figures and do not represent the only ones. The present disclosure does not specifically limit the specific forms and types of the physical devices involved.

[0045] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0046] The following is an introduction to the specific implementation of the multi-modal emergency vehicle handling method provided by the present disclosure:

[0047] Figure 2 is a flowchart of a multi-modal processing method for emergency vehicles provided by the first embodiment of the present disclosure, such as Figure 2 As shown, the multi-modal method for handling emergency vehicles specifically includes:

[0048] S201: Determine the travel direction of the emergency vehicle and its relative position to the own vehicle based on the collected audio data.

[0049] In this step, emergency vehicles refer to vehicles that perform specific tasks in emergency situations and have the right of way. Emergency vehicle types generally include: ambulances used for medical emergency and patient transfer, equipped with medical equipment and professional medical staff; fire trucks used for fire fighting and rescue operations, equipped with fire fighting equipment, rescue tools and firefighters; police cars used for law enforcement, patrolling and emergency response, equipped with communication equipment and police equipment; and engineering rescue vehicles used to respond to emergency engineering accidents and natural disasters, equipped with professional search and rescue tools and professional engineering technicians. When performing their tasks, emergency vehicles usually use sirens and flashing lights to alert other road users in order to obtain the right of way.

[0050] The audio collection device mounted on an ordinary vehicle (that is, the vehicle itself) collects audio data of the driving environment of the vehicle in real time, and the audio data includes audio collected by multiple audio collection devices set at different positions of the vehicle. The location where the audio collection device is set can be the top of the vehicle, the outside of the door, or around the rearview mirror of the vehicle; the audio collection device, as described in the scenario embodiment, can be a microphone, a sound pressure sensor, a sound wave sensor, and other electronic devices that can capture and process various sound signals in the environment; the number of audio collection devices can be 3 or more. For example, in an optional manner, an audio collection device can be set at each of the four corners on the top of the vehicle or around the vehicle body, and an audio array can be formed when audio collection is performed. For the type, number, and setting position of the audio collection device, in actual applications, it is necessary to appropriately adjust according to the size and shape of the vehicle, cost and technical limitations, and algorithm requirements, and the present disclosure does not make specific limitations.

[0051] The direction of travel of the emergency vehicle describes the relative movement direction between the emergency vehicle and the ego vehicle, including moving away from and towards each other. When the emergency vehicle is moving away from the ego vehicle, that is, the distance between the emergency vehicle and the ego vehicle gradually increases or first decreases and then increases, it indicates that the emergency vehicle is moving away from the ego vehicle or first approaches and then moves away from the ego vehicle, and the emergency vehicle and the ego vehicle are traveling in lanes in different directions. When the emergency vehicle is moving towards the ego vehicle, that is, the distance between the emergency vehicle and the ego vehicle gradually decreases, it indicates that the emergency vehicle is approaching the ego vehicle, and the emergency vehicle and the ego vehicle are traveling in lanes in the same direction. Relative position refers to the spatial relationship between the ego vehicle and the emergency vehicle, which is usually based on the lateral axis of the ego vehicle, that is, the center axis of the vehicle body, and divides the vehicle space into two areas: the front and the rear.

[0052] Specifically, it can be understood that when the distance between the sound source and the audio collection device is far, the sound intensity of the audio data collected by the audio collection device is low. Conversely, when the distance between the sound source and the audio collection device is close, the sound intensity of the audio data collected by the audio collection device is high. Based on the collected audio data, if the sound intensity of the audio data segment is getting lower and lower, it is determined that the emergency vehicle is moving away from the vehicle or the emergency vehicle is moving towards the vehicle, located in the area in front of the vehicle and moving away from the vehicle; if the sound intensity of the audio data segment is getting higher and higher, it is determined that the emergency vehicle is moving towards the vehicle, located in the area behind the vehicle and approaching the vehicle.

[0053] S202: Determine whether to collect image data of the emergency vehicle according to the relative position.

[0054] In this step, the relative position refers to the spatial relationship between the vehicle and the emergency vehicle. It is usually based on the lateral axis of the vehicle, that is, the center axis of the vehicle body, and the space of the vehicle is divided into two areas: the front area and the rear area. The front area refers to the space extending forward from the front bumper of the vehicle, that is, any vehicle or object located in front of the axis is considered to be in the front area of ​​the vehicle; the rear area refers to the space extending backward from the rear bumper of the vehicle, that is, any vehicle or object located behind the axis is considered to be in the rear area of ​​the vehicle.

[0055] Specifically, if the emergency vehicle is located in the area in front of the vehicle, there is no need to start the image acquisition device to collect image data of the emergency vehicle. At this time, the emergency vehicle in the area in front of the vehicle is already within the driver's field of vision, and the behavior and path decision of the emergency vehicle have little impact on the driving decision of the vehicle. The vehicle does not need to make an evasive move, that is, it is determined that the image data of the emergency vehicle does not need to be collected. At this time, the vehicle only needs to continue to obey traffic rules and maintain a safe distance. If the emergency vehicle is located in the area behind the vehicle, indicating that the emergency vehicle may be approaching the vehicle, the image acquisition device is started to collect image data of the emergency vehicle, that is, it is determined that the image data of the emergency vehicle needs to be collected in order to make appropriate evasive moves, shorten the passage time of the emergency vehicle, and improve the passage efficiency of the emergency vehicle.

[0056] S203: When it is determined that image data needs to be collected, image data obtained by collecting images of the location where the emergency vehicle is located is obtained.

[0057] In this step, when the emergency vehicle is located in the rear area of ​​the own vehicle, it is determined that the image acquisition device needs to be started to acquire image data, and image data obtained by acquiring images of the position where the emergency vehicle is located is obtained.

[0058] Specifically, the image acquisition device disposed at the rear of the vehicle is controlled to acquire images from the rear of the vehicle to obtain image data. As described in the scenario embodiment, the image acquisition device may be an electronic device such as a camera, an infrared camera, a stereoscopic vision camera, etc. that can capture and process visual information in the environment; the image acquisition device may be disposed at the rear of the vehicle, such as near the rear windshield, on the trunk, or on the parking bumper, etc., which can effectively capture dynamic information of the rear area; the number of image acquisition devices may be one or more, a single image acquisition device may provide a basic rearward field of view, and multiple image acquisition devices may provide a wider field of view coverage, and may also achieve redundancy and fault tolerance to ensure that the acquisition of image data is not delayed when a certain device fails. The type, number, and location of the image acquisition device may be appropriately adjusted in actual applications according to the required coverage, accuracy requirements, cost and technical limitations, and algorithm requirements, and the present disclosure does not make specific limitations.

[0059] S204: Determine the lane relationship between the lane where the vehicle is located and the lane where the emergency vehicle is located based on the image data.

[0060] In this step, in actual application scenarios, not all emergency lanes need to be avoided, such as emergency vehicles that are not in the vehicle's lane. Therefore, after the image acquisition device acquires the image data, it is necessary to identify and process the image data to identify the lane in which the vehicle is traveling and the lane in which the emergency vehicle is traveling. Through the features in the image data, it is analyzed whether the lane in which the vehicle is located and the lane in which the emergency lane is located are the same lane. The lane relationship in this solution is used to indicate whether the lane in which the vehicle is located and the lane in which the emergency lane is located are the same lane. In the subsequent emergency avoidance decision-making process, further decisions need to be made based on the lane relationship.

[0061] S205: Push an avoidance strategy for the emergency vehicle based on the lane relationship, driving direction and relative position.

[0062] In this step, the lane relationship, driving direction and relative position analyzed based on the image data obtained by image acquisition of the emergency vehicle's location together indicate the specific location and driving intention of the emergency vehicle to decide whether the emergency vehicle needs to be avoided. If the emergency vehicle needs to be avoided, a more accurate avoidance strategy for the emergency vehicle is pushed to shorten the emergency vehicle's passage time.

[0063] For example, in the case of multiple lanes, the lane relationship indicates that the emergency vehicle and the vehicle are in the same lane, and the driving direction and relative position together indicate that the emergency vehicle is approaching the vehicle, then the avoidance strategy for the emergency vehicle needs to be pushed. If the emergency vehicle and the vehicle are not in the same lane, or they are in the same lane but the emergency vehicle is far away from the vehicle, then there is no need to push the avoidance strategy for the emergency vehicle for the time being.

[0064] The multimodal emergency vehicle processing method provided by the embodiment of the present disclosure realizes accurate detection and response to emergency vehicles by combining audio data and image data. Based on the audio data in the environment collected in real time by multiple audio acquisition devices installed on the vehicle, the driving direction and relative position of the emergency vehicle are determined. When it is determined that the emergency vehicle is located in the rear area of ​​the vehicle, the image acquisition device of the vehicle is started to collect image data of the emergency vehicle. The lane relationship is analyzed based on the image data, and finally, according to the specific position and driving intention of the emergency vehicle provided by the lane relationship, driving direction and relative position, a more accurate avoidance strategy is pushed, such as suggesting lane change in multi-lane situations, so as to provide priority passage space for emergency vehicles. The above method helps to shorten the passage time of emergency vehicles and improve the passage efficiency of emergency vehicles, thereby better ensuring public safety and service response capabilities in emergency situations, and significantly improving the vehicle's perception and response capabilities to emergencies.

[0065] Figure 3 is a flowchart of a multi-modal method for handling emergency vehicles provided by the second embodiment of the present disclosure, such as Figure 3 As shown, based on the first embodiment described above, the driving direction of the emergency vehicle and its relative position to the vehicle are determined based on the collected audio data, specifically including:

[0066] In a specific implementation, sound frequency features are extracted from audio data, and whether an emergency vehicle is present is determined based on the sound frequency features.

[0067] In a specific implementation of the present disclosure, the following example is taken as an example of four microphones mounted on the top of the vehicle, which are located at the four corners of the top of the vehicle to form a symmetrical array layout, and real-time audio data of the driving environment of the vehicle is collected. To adapt to the alarm sound frequency of emergency vehicles, the sampling frequency of each microphone is set to 48KHZ, and the signal coverage range is 100HZ to 10KHZ. Based on the audio data obtained by the microphone, the sound frequency characteristics are extracted.

[0068] The frequency characteristic of sound refers to the number of times the sound vibrates per second. The frequency range is generally [10 -4,20000]HZ, the frequency of emergency vehicle alarm sound is generally [400,2000]HZ. The frequency of police car alarm sound is generally [700,1500]HZ, and the tone mode is usually "wail" (long and slow rise and fall) and "yelp" (fast rise and fall) and "piercer" or "hi-lo" (alternating high and low pitch); the frequency of ambulance alarm sound is generally [500,1400]HZ, and the tone mode is usually "wail" and "yelp", which changes slightly slower than the police car alarm; the frequency of fire truck alarm sound is generally [400,1200]HZ, and the tone mode is usually "wail" and "hi-lo". Different sound frequency and tone characteristics are designed to ensure that emergency vehicles can be quickly identified in complex traffic environments.

[0069] In a specific implementation, after determining that an emergency vehicle exists, the direction of travel of the emergency vehicle and its relative position to the vehicle are determined based on the sound frequency characteristics of the emergency vehicle in each audio frequency.

[0070] As described in the first embodiment of the present disclosure, the direction of travel of the emergency vehicle describes the relative movement direction between the emergency vehicle and the own vehicle, including moving away from and towards each other; the relative position refers to the spatial relationship between the own vehicle and the emergency vehicle, which is usually based on the lateral axis of the own vehicle, that is, the center axis of the vehicle body, and divides the space of the vehicle into two areas, the front and the rear. Based on the sound frequency characteristics extracted from the audio data, it is determined whether the emergency vehicle and the own vehicle are moving away from or towards each other, and it is determined whether the emergency vehicle is located in the front area or the rear area of ​​the own vehicle. By clarifying the direction of travel of the emergency vehicle and the relative position of the own vehicle, a more accurate avoidance strategy can be determined to shorten the time for the emergency vehicle to perform rescue missions.

[0071] Specifically, if the sound frequency of the emergency vehicle in each audio is getting smaller and smaller, that is, the sound intensity of this segment of audio data is getting lower and lower, then it is determined that the emergency vehicle is moving away from the own vehicle or the emergency vehicle is moving towards the own vehicle, is located in the area in front of the own vehicle and is moving away from the own vehicle; if the sound frequency of the emergency vehicle in each audio is getting larger and larger, that is, the sound intensity of this segment of audio data is getting higher and higher, then it is determined that the emergency vehicle is moving towards the own vehicle, is located in the area behind the own vehicle and is approaching the own vehicle.

[0072] Extracting sound frequency features from audio data and determining whether there is an emergency vehicle based on the sound frequency features specifically includes the following steps:

[0073] S301: For any audio in the audio data, perform noise filtering and automatic gain processing on the audio to obtain processed audio.

[0074] In this step, the audio collection device collects audio data from the driving environment of the vehicle in real time. The collected audio data may contain various noises, such as wind, rain, thunder, engine, horn, conversations of passengers in the vehicle, music, and construction equipment noise when the vehicle passes through the construction area, in addition to the sirens of emergency vehicles. In order to improve the instructions and usability of the audio data, for any audio in the audio data, the noise is filtered out and passed through a bandpass filter.

[0075] Specifically, the noise filtering method includes adaptive noise suppression algorithm, Kalman filter algorithm, Wiener filter algorithm and deep learning methods. In practical applications, it needs to be adjusted appropriately according to different audio signal characteristics, computing resources and real-time requirements. Then the audio data with noise filtered out is passed through a bandpass filter, that is, audio signals in the frequency range of [400,2000]HZ are allowed to pass through, and audio signals outside this range are suppressed or attenuated, which further effectively reduces the interference of environmental noise, improves the clarity and quality of audio data, and enhances the availability of audio data.

[0076] In addition, when the audio acquisition device collects audio data from the driving environment of the vehicle in real time, the distance between the sound source and the audio acquisition device will significantly affect the strength of the captured audio signal. When the distance between the sound source and the audio acquisition device is far, the signal received by the audio acquisition device is relatively weak and difficult to be effectively identified. On the contrary, when the distance between the sound source and the audio acquisition device is close, the signal received by the audio acquisition device is relatively strong. If it exceeds the dynamic range that the audio acquisition device can receive, it may cause signal distortion and clipping. In order to balance the strength of the audio data signal and make the subsequent audio data signal processing more stable and accurate, the audio is automatically gain processed to obtain the processed audio, ensuring that the audio signal can be maintained in a suitable dynamic range at different intensities, thereby avoiding the low signal being ignored or the high signal causing distortion.

[0077] S302: Convert the processed audio from a time domain signal to a frequency domain signal to obtain a converted audio signal.

[0078] In this step, the time domain signal of the audio data represents the change of the amplitude of the audio signal over time, and the frequency domain signal represents the frequency components and amplitude of the audio signal, which is suitable for analyzing the spectral characteristics of the signal, such as frequency distribution, harmonics, bandwidth, etc., and is convenient for identifying and processing signals of specific frequencies. In order to more effectively analyze the processed audio data, the processed audio is converted from the time domain signal to the frequency domain signal to obtain the converted audio signal.

[0079] Specifically, in order to analyze the audio data more accurately, the processed audio is divided into small fixed-length time windows in the time domain, usually 20 to 40 milliseconds. For example, when the sampling frequency is 48KHZ and the time window is 40 milliseconds, the time window contains 1920 audio data sampling points. Then, in order to reduce the spectrum leakage of the audio data signal, each time window is windowed. Commonly used window functions include Hamming window, rectangular window, Gaussian window, and Blackman window. When selecting the window function, it is necessary to decide according to the specific application scenario and analysis requirements. For example, if better frequency resolution is required, a wider window function is selected; if spectrum leakage needs to be reduced, a window function with smooth edges is selected. Finally, a fast Fourier transform is performed on each time window to convert the processed audio from the time domain to the frequency domain signal to obtain the converted audio signal. The time domain signal of each signal segment is decomposed into amplitude and phase information to more accurately find the frequency range of the emergency vehicle alarm sound.

[0080] S303: extracting frequency features of the converted audio signal to obtain sound frequency features.

[0081] In this step, in order to focus more on the most informative part of the audio signal and improve the efficiency and accuracy of subsequent analysis and processing, the frequency feature extraction of the converted audio signal is performed for further analysis and processing. The frequency feature extraction method generally includes wavelet transform, peak detection and envelope detection, etc., which aims to extract the most informative part from the converted audio signal, thereby improving the efficiency and accuracy of subsequent analysis and processing.

[0082] In a specific implementation of the present disclosure, peak detection is performed for each signal segment in the converted audio signal to identify and locate the local maximum in a segment of the audio signal, that is, the portion of the audio signal that contains important features, and the frequency with the largest amplitude in each signal segment is determined as the main frequency, and the harmonics of the main frequency of each signal segment are obtained. For example, if the main frequency of the signal segment is 800 Hz, the spectrum amplitudes that appear at 1600 Hz and 2400 Hz are taken as harmonics. According to the main frequency and harmonics of each signal segment, the sound frequency characteristics are obtained, including the main frequency and its harmonics.

[0083] S304: Determine whether there is an emergency vehicle based on the sound frequency characteristics and the frequency characteristics of multiple alarm sounds acquired in advance.

[0084] In this step, the frequency characteristics of the multiple alarm sounds acquired in advance include typical frequency characteristics of emergency vehicles, including the main frequency and its harmonics. Assuming that the main frequency of the fire truck alarm is 800HZ, its harmonics will appear in the spectrum amplitude at 1600HZ and 2400HZ; the main frequency of the ambulance alarm is 1000HZ, then its harmonics will appear in the spectrum amplitude at 2000HZ and 3000HZ; the main frequency of the police car alarm is 700HZ, then its harmonics will appear in the spectrum amplitude at 1400HZ and 2100HZ.

[0085] Specifically, the type of the emergency vehicle is determined based on the sound frequency characteristics and the frequency characteristics of multiple alarm sounds acquired in advance; the vehicle type includes any one of an ambulance, a fire truck, a police car, and an engineering rescue vehicle. In a specific implementation of the present disclosure, by comparing the sound frequency characteristics of the audio signal with the frequency characteristics of the alarm sound acquired in advance, if the main frequency and harmonics of the audio signal segment match the main frequency and harmonics of the alarm sound acquired in advance, it is determined that an emergency vehicle exists, and the type of the emergency vehicle can be determined at the same time.

[0086] The multimodal processing method for emergency vehicles provided by the embodiment of the present disclosure identifies whether an emergency vehicle exists by extracting sound frequency features from audio data. First, the collected audio data is subjected to noise filtering and automatic gain processing to improve the clarity and quality of the audio signal. The processed audio signal is converted from the time domain to the frequency domain so as to more effectively analyze its spectral features. In the frequency feature extraction stage, the peak detection method is used to identify the local maximum in the audio signal and determine the main frequency and harmonics of the signal segment. By comparing the main frequency and harmonics in multiple signal segments, the sound frequency features of the audio signal are extracted. Finally, the extracted sound frequency features are compared with the frequency features of the emergency vehicle alarm sound obtained in advance to determine whether an emergency vehicle exists, and the type of vehicle can be further identified. Through the above method, the accuracy of identifying emergency vehicles is improved, so as to provide timely warning information to vehicle drivers and improve road traffic safety.

[0087] Figure 4 is a flowchart of a method for handling emergency vehicles based on multimodality provided in the third embodiment of the present disclosure, such as Figure 4 As shown, based on the above embodiment, in determining the direction of travel of the emergency vehicle and the relative position to the vehicle according to the sound frequency characteristics of the emergency vehicle in each audio, the following steps are specifically included:

[0088] S401: According to the position of each audio collection device in the vehicle and the sound frequency characteristics of the emergency vehicle in each audio, obtain the time difference when different audio collection devices receive the sound of the emergency vehicle and / or the intensity difference when different audio collection devices receive the sound of the emergency vehicle at the same time.

[0089] In this step, the time difference of arrival (TDOA) determines the location of the sound source based on the time difference of the sound wave propagating between different sensors. Multiple audio collection devices are arranged in the vehicle, and the positions are known. When the sound source makes a sound, the sound waves will reach multiple audio collection devices at different times. By accurately measuring the time difference, the direction and distance of the sound source relative to the direction array of the audio collection device can be calculated. Intensity difference of localization (IDOL) uses the intensity difference of sound at different locations to infer the location of the sound source. It can be understood that the intensity of the sound decreases as the distance between the audio collection device and the sound source increases.

[0090] Specifically, the time difference between different audio collection devices receiving the sound of the emergency vehicle can be expressed by the following formula:

[0091]

[0092] Among them, τ ij Indicates the time difference between audio acquisition devices i and j receiving the sound source signal, in seconds; x i t represents the signal value received by the audio acquisition device i at time t, which is the result of converting the sound source signal received by the audio acquisition device into a digital signal; x j (t+τ) represents the signal value received by the audio acquisition device j at time t.

[0093] The relative position of the emergency vehicle can be further determined by obtaining the time difference when the sound of the emergency vehicle is received by different audio collection devices or the intensity difference when the sound of the emergency vehicle is received by different audio collection devices at the same time.

[0094] S402: Determine the relative position of the emergency vehicle and the vehicle according to the time difference between different audio collection devices receiving the sound of the emergency vehicle and / or the intensity difference between different audio collection devices receiving the sound of the emergency vehicle at the same time.

[0095] In the specific implementation of the present disclosure, the four microphones mounted on the top of the vehicle are located at the four top corners of the top of the vehicle, forming a symmetrical array layout to collect audio data of the driving environment of the vehicle in real time. The four microphones are M1 located at the upper left corner, M2 at the lower left corner, M3 at the upper right corner, and M4 at the lower right corner.

[0096] The four microphones collect audio data of the driving environment of the vehicle in real time. If microphone M1 or microphone M3 receives the sound of the emergency vehicle first, followed by microphone M2 or microphone M4, it indicates that the emergency vehicle is in front of the vehicle; if microphone M2 or microphone M4 receives the sound of the emergency vehicle first, followed by microphone M1 or microphone M3, it indicates that the emergency vehicle is behind the vehicle. Alternatively, at the same time t, if the intensity of the sound of the emergency vehicle received by microphone M1 or microphone M3 is greater than the intensity of the sound of the emergency vehicle received by microphone M2 or microphone M4, it indicates that the emergency vehicle is in front of the vehicle; if the intensity of the sound of the emergency vehicle received by microphone M1 or microphone M3 is less than the intensity of the sound of the emergency vehicle received by microphone M2 or microphone M4, it indicates that the emergency vehicle is behind the vehicle.

[0097] In a specific implementation, after the relative position of the emergency vehicle and the vehicle is determined, the relative distance and speed between the emergency vehicle and the vehicle can be estimated by the dynamic change of the sound intensity of the emergency vehicle. The relative distance estimation formula is as follows:

[0098]

[0099] Where f′ is the frequency of the sound of the emergency vehicle received by the ego vehicle; f is the fundamental frequency of the sound of the emergency vehicle, that is, the original frequency of the sound source of the emergency vehicle without any relative motion; v represents the speed of sound, which is about 343 m / s in air, depending on temperature and other factors; v0 represents the audio acquisition device, that is, the speed of the ego vehicle relative to the air. If the audio acquisition device is stationary, v0 = 0; v S Represents the speed of the source of the sound, i.e. the emergency vehicle, relative to the air.

[0100] The sound source, i.e., the speed v of the emergency vehicle relative to the air, is obtained according to the above formula by using the fundamental frequency of the sound of the emergency vehicle and the frequency of the sound collected by the audio collection device of the vehicle within a period of time, such as 30 seconds, and the speed of the vehicle. S , and further calculate the distance between the emergency vehicle and the vehicle.

[0101] S403: Determine the driving direction of the emergency vehicle according to the change of the sound intensity of the emergency vehicle collected by any audio collection device at adjacent moments, where the driving direction includes approaching the own vehicle or moving away from the own vehicle.

[0102] In this step, the direction of travel of the emergency vehicle describes the relative movement direction between the emergency vehicle and the vehicle, and the direction of travel includes approaching or moving away from the vehicle, that is, the emergency vehicle and the vehicle are moving towards or away from each other. As described in the first embodiment S201 of the present disclosure: It can be understood that when the distance between the sound source and the audio acquisition device is far, the sound intensity of the audio data collected by the audio acquisition device is low, and conversely, when the distance between the sound source and the audio acquisition device is close, the sound intensity of the audio data collected by the audio acquisition device is high.

[0103] Therefore, if the sound intensity of the emergency vehicle collected by any audio collection device at adjacent moments becomes lower and lower, it is determined that the emergency vehicle is moving away from the own vehicle or the emergency vehicle is moving towards the own vehicle, is located in the area in front of the own vehicle and is moving away from the own vehicle; if the sound intensity of the emergency vehicle collected by any audio collection device at adjacent moments becomes higher and higher, it is determined that the emergency vehicle is moving towards the own vehicle, is located in the area behind the own vehicle and is approaching the own vehicle.

[0104] The multimodal processing method for emergency vehicles provided in the embodiments of the present disclosure realizes the detection and positioning of emergency vehicles by utilizing multiple audio acquisition devices (such as microphones) installed in the vehicle. The method determines the relative position of the emergency vehicle and the vehicle by analyzing the time difference or intensity difference of the emergency vehicle sound received by the audio acquisition device. In addition, by analyzing the dynamic changes in the sound intensity, the direction of travel of the emergency vehicle is determined, that is, whether it is approaching or moving away from the vehicle. Through the above method, the multimodal processing method can effectively identify and locate emergency vehicles in complex traffic environments, realize high-precision detection and positioning of emergency vehicles, and improve the safety of the vehicle and its ability to respond to emergencies.

[0105] Figure 5 is a flowchart of a method for handling emergency vehicles based on multimodality provided in the fourth embodiment of the present disclosure, such as Figure 5 As shown, based on the above embodiment, determining the lane relationship between the lane where the vehicle is located and the lane where the emergency vehicle is located according to the image data specifically includes the following steps:

[0106] S501: Extracting a red and blue light flickering area from the image data based on a deep learning algorithm, and obtaining a light source point of the red and blue light flickering area.

[0107] In this step, the light source point in the red and blue flashing area refers to the warning lights installed on emergency vehicles such as police cars, ambulances, fire trucks, and engineering rescue vehicles. The warning lights are usually composed of multiple LED lights that can quickly switch colors and brightness to achieve red and blue light flashing alternately. Since red and blue are a combination of strong contrasting colors, they can provide high visibility under various lighting conditions. Emergency vehicles use red and blue flashing warning lights to attract the attention of other road users so that they can have priority in emergency situations.

[0108] Specifically, when the emergency vehicle is located in the rear area of ​​the vehicle, it is determined that the image acquisition device needs to be started to collect image data, and the image data obtained by collecting images of the position of the emergency vehicle is obtained. When the sound intensity of the emergency vehicle collected by any audio acquisition device at adjacent moments becomes higher and higher, it is determined that the emergency vehicle is traveling in the opposite direction of the vehicle, is located in the rear area of ​​the vehicle and is approaching the vehicle. At this time, based on the YOLO detection algorithm, the red and blue light flashing area is identified from the image data, and the light source point of the red and blue light flashing area is obtained. The YOLO detection algorithm can accurately locate the area where the flashing lights are located, and provide a target frame, that is, the lane where the target frame of the light source point of the red and blue light flashing area is located may be the location of the emergency vehicle. Further, the brightness and color intensity of each pixel in the red and blue light flashing area are obtained, and based on the brightness and color intensity of each pixel, the light source point in the red and blue light flashing area is determined. The red and blue light flashing area may cover a large area, and it is impossible to determine the specific location information of the emergency vehicle. By obtaining the brightness and color intensity of each pixel, the point with the largest brightness and color intensity of each pixel is determined to be the point with the strongest light source, that is, the specific location of the emergency vehicle, thereby achieving effective position detection of the emergency vehicle.

[0109] It should be noted that the YOLO detection algorithm here is only used as an example. In actual applications, it may also be a target detection algorithm based on deep learning such as FasterRCNN and SSD. In actual applications, it can be appropriately adjusted according to specific needs and environmental conditions, and this disclosure does not make specific limitations.

[0110] S502: Perform lane recognition on the image data to determine lane information.

[0111] In this step, the image data collected in real time by the image acquisition device usually includes multiple lane information, road sign information, cyclists, pedestrians, vehicles and other road features.

[0112] Specifically, the image data is first preprocessed, including denoising, grayscale conversion, and histogram equalization, to improve the accuracy of subsequent lane recognition. Then, the edge detection algorithm is used to preliminarily determine the possible lane line positions. In order to further accurately identify the lane lines, image processing techniques such as Hough transform are used to extract the lines that meet the lane line characteristics from the edge detection results, determine the lane information, and provide support for subsequent vehicle control and decision-making.

[0113] S503: Determine the lane relationship based on the lane information and the light source points in the red and blue light areas.

[0114] The lane relationship is used to indicate whether the lane where the emergency vehicle is located is the same lane as the lane where the vehicle is located.

[0115] In this step, the location of the light source point in the red and blue light area can be considered as the approximate location of the emergency vehicle. The lane where the light source point is located can be determined in combination with the lane information, that is, the lane where the emergency vehicle is located can be determined. Combined with the viewing angle position of the captured image in the image data, it is comprehensively determined whether the lane where the emergency vehicle and the vehicle are traveling are in the same lane, and the lane relationship is obtained.

[0116] Accordingly, according to the lane relationship, the driving direction and the relative position, pushing the avoidance strategy for the emergency vehicle can be implemented as follows:

[0117] S504: If the emergency vehicle and the own vehicle are traveling in the same lane, and the emergency vehicle is traveling towards the own vehicle, an avoidance strategy for the emergency vehicle is pushed based on the relative position.

[0118] In this step, if the emergency vehicle and the self-vehicle are not in the same lane, it can be considered that the emergency vehicle and the self-vehicle are traveling in different directions or in different lanes, and the self-vehicle has little impact on the passage of the emergency vehicle. In this case, the self-vehicle only needs to continue to obey traffic rules and keep a safe distance. If the emergency vehicle and the self-vehicle are in the same lane, it is necessary to comprehensively judge whether to avoid the emergency vehicle based on the relative position of the emergency vehicle and the self-vehicle and the direction of travel of the emergency vehicle. If avoidance is required, the avoidance strategy for the emergency vehicle is pushed.

[0119] Specifically, if the emergency vehicle is in the same lane as the self-vehicle, and the emergency vehicle is far away from the self-vehicle, at this time, the self-vehicle has little impact on the passage of the emergency vehicle, so the self-vehicle only needs to continue to obey traffic rules and maintain a safe distance. If the emergency vehicle is in the same lane as the self-vehicle, and the emergency vehicle is approaching the self-vehicle, and the distance from the self-vehicle is less than the preset distance, for example, less than one hundred meters, at this time, the self-vehicle has a greater impact on the passage of the emergency vehicle, and the avoidance strategy for the emergency vehicle needs to be pushed in time to ensure that the emergency vehicle can pass smoothly. Specifically, this step S504 can be implemented as the following steps S5041-S5043.

[0120] S5041: Determine the distance between the emergency vehicle and the vehicle based on the relative position.

[0121] In this step, after determining that the emergency vehicle and the lane in which the vehicle is traveling are in the same lane and the emergency vehicle is located in the rear area of ​​the vehicle, the distance between the emergency vehicle and the vehicle needs to be determined. Specifically, according to the method provided in the third embodiment S402, the distance between the emergency vehicle and the vehicle is determined. The implementation principle and technical effect are similar and will not be repeated here. When the emergency vehicle is far away from the vehicle, the vehicle has enough time and space to make appropriate driving adjustments; when the emergency vehicle is close to the vehicle, the vehicle needs to take immediate action to avoid obstructing the passage of the emergency vehicle. Determining the distance between the emergency vehicle and the vehicle helps to avoid potential collision risks and reduce traffic congestion caused by the passage of emergency vehicles.

[0122] S5042: If the distance is less than a preset distance threshold, an avoidance strategy is determined based on the road conditions on which the vehicle is traveling; the avoidance strategy includes a pull-over avoidance strategy or a lane change avoidance strategy.

[0123] In this step, the preset distance threshold determines when to start the avoidance strategy. The setting of this threshold directly affects the reaction time and decision-making process of the driver or the autonomous driving system. Therefore, the setting of the preset distance threshold needs to be based on various factors such as the minimum safe distance that other vehicles should maintain when the emergency vehicle approaches as stipulated by traffic regulations, road conditions, environmental factors, and the braking performance of the vehicle. Through a reasonable preset distance threshold, it can be ensured that the driver or the autonomous driving system adopts a reasonable avoidance strategy in an emergency.

[0124] The preset distance threshold value range can be set to [50, 100], and the specific value can be 50, 60, 70, 80, 90, 100, etc., that is, when the distance between the emergency vehicle and the vehicle is 50 meters, 60 meters, 70 meters, 80 meters, 90 meters, 100 meters, the pull-over avoidance strategy or lane change avoidance strategy is pushed based on the road conditions. The preset distance threshold value here is only used for example, and the specific value needs to be appropriately adjusted according to the actual application situation, and this disclosure does not limit it.

[0125] S5043: Push the avoidance strategy through at least one output interface of the vehicle terminal, the vehicle computer interface or the speaker.

[0126] In this step, the vehicle terminal refers to a dedicated device installed in the vehicle, such as a navigation system or infotainment system, which can display detailed avoidance strategies on the screen and provide visual navigation assistance. The vehicle computer interface refers to the vehicle's central control display screen, which can intuitively display the avoidance path, the location of the emergency vehicle, etc. The speaker, as another output interface, can convey the avoidance strategy through voice prompts, which can reduce the risk of driver distraction and ensure that the driver can quickly understand and take action.

[0127] Through the above output interface, the avoidance strategy can be effectively conveyed to the driver, ensuring that the avoidance strategy can be executed quickly and accurately in emergency situations. This not only improves the efficiency of information transmission, but also enhances the driver's response ability, thereby improving overall road safety.

[0128] The multimodal emergency vehicle processing method provided by the disclosed embodiment first uses a deep learning algorithm to extract the red and blue light flashing area from the image data and identify the specific location of the emergency vehicle. Then, lane recognition is performed through image processing technology to determine whether the emergency vehicle and the vehicle are in the same lane. When it is confirmed that the emergency vehicle is behind the vehicle and in the same lane, the corresponding avoidance strategy is pushed according to the relative position and distance. The avoidance strategy is communicated to the driver through output interfaces such as the on-board terminal, the vehicle interface or the speaker to ensure that he can take action in time. Through the above method, it is ensured that the driver or the automatic driving system can respond quickly in an emergency, reducing the potential risk of collision and traffic congestion, and improving the accuracy of emergency vehicle detection and the effectiveness of the avoidance strategy, thereby improving overall road safety and traffic efficiency.

[0129] Figure 6 is a flowchart of a multi-modal processing method for emergency vehicles provided in the fifth embodiment of the present disclosure, such as Figure 6 As shown, based on the above embodiment, in determining the avoidance strategy based on the road conditions of the vehicle, the following steps are specifically included:

[0130] S601: If the road condition indicates that there are no other drivable roads on both sides of the vehicle, the avoidance strategy is determined to be a pull-over avoidance strategy. The avoidance strategy also includes the vehicle type of the emergency vehicle.

[0131] In this step, the road conditions are obtained by detecting the image information or navigation information around the vehicle. The road conditions indicate that there are no other drivable roads on both sides of the vehicle, including: the lane where the vehicle is traveling is a one-way lane or the lane where the vehicle is traveling is a multi-lane, but other lanes are occupied by other road users, and the vehicle cannot complete the lane change avoidance. At this time, the vehicle needs to adopt a sideways avoidance strategy to complete the avoidance of the emergency vehicle. At the same time, according to the method provided in S304 of the second embodiment, when it is determined that there is an emergency vehicle, the type of the emergency vehicle is also determined. Therefore, the avoidance strategy also includes the vehicle type of the emergency vehicle.

[0132] Specifically, the pull-over avoidance strategy includes: when the lane where the vehicle is traveling is full of vehicles, activate the right turn signal, decelerate to a speed lower than the preset speed, and then drive close to the edge of the right lane after deceleration; when the lane where the vehicle is traveling is a one-way lane, activate the right turn signal, decelerate to a speed lower than the preset speed, and then stop close to the edge of the right lane after deceleration. After the emergency vehicle passes, the vehicle must continue driving immediately and maintain a safe distance.

[0133] The preset speed refers to the safe speed to which the vehicle needs to slow down when executing the pull-over avoidance strategy to ensure safety and reliability when avoiding emergency vehicles. The preset speed setting must be based on the minimum speed limit requirements of the lane specified by traffic regulations and standards, road conditions, and environmental factors to ensure that the emergency vehicle can be effectively given way during the avoidance process while maintaining vehicle stability and safety.

[0134] Specifically, when the lane your vehicle is traveling in is a one-way lane and the distance between the emergency vehicle and your vehicle is less than the preset threshold of 50 meters, the system voice prompts the driver to turn on the right turn signal and decelerate to 15km / h, and then stop near the edge of the right lane after deceleration. At the same time, a yellow warning box on the central control display screen prompts: Ambulance behind, pull over to give way. If the distance between the emergency vehicle and your vehicle is less than 30 meters, your vehicle will also execute an avoidance strategy, and a red warning box on the central control display screen prompts: Ambulance behind, pull over to give way.

[0135] S602: If the road condition indicates that there are other drivable roads on both sides of the vehicle, the avoidance strategy is determined to be a lane change avoidance strategy.

[0136] In this step, the road conditions are obtained by detecting the image information or navigation information around the vehicle. The road conditions indicate that there are other drivable roads on both sides of the vehicle, indicating that the lane where the vehicle is driving is a multi-lane lane, and there are no other road users driving in other lanes for the time being. The vehicle can complete the lane change avoidance. At this time, the vehicle needs to adopt a lane change avoidance strategy to complete the avoidance of the emergency vehicle. Specifically, the lane change avoidance strategy includes: changing the vehicle to other drivable lanes.

[0137] S6021: By detecting the image information or navigation information around the vehicle, identifying the vehicles in other drivable lanes, and determining the target lane; the lane change avoidance strategy specifically includes: changing the vehicle to the target lane for driving.

[0138] In this step, the road condition is obtained by detecting the image information or navigation information around the vehicle. The road condition indicates that there are other specific target lanes on both sides of the vehicle. If the target lane is on the right side of the vehicle's driving lane, the right turn signal is activated to change the vehicle to the right lane for driving; if the target lane is on the left side of the vehicle's driving lane, the left turn signal is activated to change the vehicle to the left lane for driving.

[0139] Specifically, when the lane where the vehicle is traveling is multi-lane, and there are other specific target lanes on both sides of the vehicle, and the distance between the emergency vehicle and the vehicle is less than the preset threshold of 50 meters, the system voice prompts the driver that the target lane is on the right side of the lane where the vehicle is traveling, activates the right turn signal, and changes the vehicle to the right lane for driving. At the same time, the yellow warning box on the central control display screen prompts: Ambulance behind, change lanes to give way. If the distance between the emergency vehicle and the vehicle is less than 30 meters, the vehicle also executes the avoidance strategy, and the red warning box on the central control display screen prompts: Ambulance behind, change lanes to give way.

[0140] S603: Control the vehicle to avoid the emergency vehicle according to the avoidance strategy.

[0141] According to the above avoidance strategy, the vehicle avoids the emergency vehicle to ensure smooth passage of the emergency vehicle.

[0142] The multimodal method for handling emergency vehicles provided by the embodiments of the present disclosure formulates and implements effective avoidance strategies by comprehensively utilizing image data, navigation information, and vehicle driving status. Specifically, the road conditions around the vehicle are first detected to determine whether there are any drivable lanes. When there are no other drivable roads on both sides of the vehicle, a pull-over avoidance strategy is selected. If it is detected that there are other drivable roads on both sides of the vehicle, a lane change avoidance strategy is selected to guide the vehicle to change lanes to the target lane. Through the above method, the accuracy and efficiency of emergency vehicle detection and avoidance are improved, the driver's operating burden is reduced, and the response speed is improved, thereby improving overall road safety and traffic smoothness.

[0143] Figure 7 is a structural block diagram of a multi-modal processing device for emergency vehicles provided by an embodiment of the present disclosure, such as Figure 7 As shown, the multi-modal emergency vehicle processing device 70 includes:

[0144] The first processing unit 701 is used to determine the travel direction of the emergency vehicle and the relative position of the emergency vehicle to the own vehicle based on the collected audio data;

[0145] The second processing unit 702 is used to determine whether to collect image data of the emergency vehicle according to the relative position;

[0146] The third processing unit 703 is used to obtain image data obtained by collecting images of the location of the emergency vehicle when it is determined that image data needs to be collected;

[0147] The fourth processing unit 704 is used to determine the lane relationship between the lane where the vehicle is located and the lane where the emergency vehicle is located according to the image data;

[0148] The fifth processing unit 705 is used to push an avoidance strategy for the emergency vehicle according to the lane relationship, driving direction and relative position.

[0149] In a possible implementation, the first processing unit 701 includes:

[0150] The first processing module 7011 is used to extract sound frequency features from the audio data and determine whether there is an emergency vehicle based on the sound frequency features;

[0151] The second processing module 7012 is used to determine the travel direction of the emergency vehicle and its relative position to the vehicle according to the sound frequency characteristics of the emergency vehicle in each audio after determining the existence of the emergency vehicle.

[0152] In a possible implementation, the first processing module 7011 includes:

[0153] The first processing submodule 70111 is used to perform noise filtering and automatic gain processing on any audio in the audio data to obtain processed audio;

[0154] The second processing submodule 70112 is used to convert the processed audio from a time domain signal to a frequency domain signal to obtain a converted audio signal;

[0155] The third processing submodule 70113 extracts the frequency characteristics of the converted audio signal to obtain the sound frequency characteristics;

[0156] The fourth processing submodule 70114 determines whether there is an emergency vehicle based on the sound frequency characteristics and the frequency characteristics of multiple alarm sounds acquired in advance.

[0157] In a possible implementation, the second processing module 7012 includes:

[0158] The fifth processing submodule 70121 is used to obtain the time difference when different audio collection devices receive the sound of the emergency vehicle and / or the intensity difference when different audio collection devices receive the sound of the emergency vehicle at the same time according to the position of each audio collection device in the vehicle and the sound frequency characteristics of the emergency vehicle in each audio;

[0159] The sixth processing submodule 70122 is used to determine the relative position of the emergency vehicle and the vehicle according to the time difference between the sound of the emergency vehicle received by different audio collection devices and / or the intensity difference of the sound of the emergency vehicle received by different audio collection devices at the same time;

[0160] The seventh processing submodule 70123 is used to determine the driving direction of the emergency vehicle according to the change of the sound intensity of the emergency vehicle collected by any audio collection device at adjacent moments, and the driving direction includes approaching or moving away from the own vehicle.

[0161] In a possible implementation, the second processing unit 702 includes:

[0162] The third processing module 7021 is used to determine that it is necessary to collect image data of the emergency vehicle if the emergency vehicle is located behind the vehicle; otherwise, it is determined that it is not necessary to collect image data of the emergency vehicle.

[0163] In a possible implementation, the fourth processing unit 704 includes:

[0164] The fourth processing module 7041 is used to extract the red and blue light flickering area from the image data based on the deep learning algorithm, and obtain the light source point of the red and blue light flickering area;

[0165] The fifth processing module 7042 is used to perform lane recognition on the image data and determine lane information;

[0166] The sixth processing module 7043 is used to determine the lane relationship according to the lane information and the light source points of the red and blue light areas, wherein the lane relationship is used to indicate whether the lane where the emergency vehicle is located is the same lane as the lane where the vehicle is located;

[0167] Accordingly, the fifth processing unit 705 includes:

[0168] The seventh processing module 7051 is used to push an avoidance strategy for the emergency vehicle according to the relative position if the lane where the emergency vehicle and the own vehicle are traveling are the same lane and the emergency vehicle is traveling towards the own vehicle.

[0169] In a possible implementation, the seventh processing module 7051 includes:

[0170] The eighth processing submodule 70511 is used to determine the distance between the emergency vehicle and the vehicle according to the relative position;

[0171] The ninth processing submodule 70512 is used to determine an avoidance strategy based on the road conditions of the vehicle if the distance is less than a preset distance threshold; the avoidance strategy includes a side avoidance strategy or a lane change avoidance strategy;

[0172] The tenth processing submodule 70513 is used to push the avoidance strategy through at least one output interface of the vehicle terminal, the vehicle computer interface or the speaker.

[0173] In a possible implementation manner, the ninth processing submodule 70512 is specifically used to:

[0174] If the road condition indicates that there are no other drivable roads on both sides of the vehicle, the avoidance strategy is determined to be a pull-over avoidance strategy;

[0175] If the road conditions indicate that there are other drivable roads on both sides of the vehicle, the avoidance strategy is determined to be a lane change avoidance strategy;

[0176] The pull-over avoidance strategy includes: activating the right turn signal, decelerating to a speed lower than the preset speed, and driving close to the edge of the right lane after deceleration; or activating the right turn signal, decelerating to a speed lower than the preset speed, and stopping close to the edge of the right lane after deceleration;

[0177] The lane change avoidance strategy includes: changing the vehicle to other drivable lanes.

[0178] In a possible implementation, the multi-modal emergency vehicle processing device 70 further includes:

[0179] The sixth processing unit 706 is used to identify the vehicles in other drivable lanes by detecting the image information or navigation information around the vehicle and determine the target lane; the lane change avoidance strategy specifically includes: changing the vehicle to the target lane for driving.

[0180] In a possible implementation, the multi-modal emergency vehicle processing device 70 further includes:

[0181] The seventh processing unit 707 is used to control the vehicle to avoid the emergency vehicle according to the avoidance strategy.

[0182] In a possible implementation, the multi-modal emergency vehicle processing device 70 further includes:

[0183] The eighth processing unit 708 is used to determine the type of the emergency vehicle according to the sound frequency characteristics and the frequency characteristics of the plurality of alarm sounds acquired in advance; the vehicle type includes: any one of an ambulance, a fire truck, a police car and an engineering rescue vehicle;

[0184] The avoidance strategy also includes the vehicle type of the emergency vehicle.

[0185] In a possible implementation, the third processing unit 703 includes:

[0186] The eighth processing module 7031 is used to control the image acquisition device disposed at the rear of the vehicle to acquire image data from the rear of the vehicle.

[0187] In a possible implementation manner, the third processing submodule 70113 is specifically used to:

[0188] Perform peak detection on each signal segment in the converted audio signal, determine the frequency with the largest amplitude in each signal segment as the main frequency, and obtain the harmonics of the main frequency of each signal segment;

[0189] The sound frequency characteristics are obtained based on the main frequency and harmonics of each signal segment.

[0190] In a possible implementation manner, the fourth processing module 7041 is specifically configured to:

[0191] Based on the YOLO detection algorithm, the red and blue light flashing areas are identified from the image data;

[0192] The brightness and color intensity of each pixel in the red and blue light flickering area are obtained, and based on the brightness and color intensity of each pixel, the light source point in the red and blue light flickering area is determined.

[0193] The multi-modal emergency vehicle processing device provided in the embodiment of the present disclosure can execute the method provided in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0194] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a vehicle, a readable storage medium, and a computer program product.

[0195] According to an embodiment of the present disclosure, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the technical solution in the aforementioned method embodiment.

[0196] According to an embodiment of the present disclosure, a computer program product provided by the present disclosure includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0197] According to an embodiment of the present disclosure, the present disclosure further provides a vehicle, comprising: a vehicle body, and a plurality of audio collection devices respectively arranged at different positions of the vehicle body;

[0198] The processing device is arranged in the vehicle body, and is used to execute the method provided by the above method embodiment, and its implementation principle and technical effect are similar, and will not be repeated here. The processing device includes an audio processing unit and a vehicle control unit.

[0199] Figure 8 1 is a block diagram of an electronic device for implementing a multimodal method for handling emergency vehicles according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0200] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. The input / output (I / O) interface 80 is also connected to the bus 804.

[0201] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0202] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the avoidance method for emergency vehicles. For example, in some embodiments, the avoidance method for emergency vehicles may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the multimodal-based emergency vehicle processing method described above may be executed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the multi-modality-based emergency vehicle handling method in any other appropriate manner (eg, by means of firmware).

[0203] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0204] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0205] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0206] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0207] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0208] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0209] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0210] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multi-modal method for handling emergency vehicles, comprising: Based on the collected audio data, determine the direction of travel of the emergency vehicle and its relative position to the own vehicle; Determining whether to collect image data of the emergency vehicle according to the relative position; When it is determined that image data needs to be collected, image data obtained by collecting images of the location of the emergency vehicle is obtained; Determining a lane relationship between a lane where the vehicle is located and a lane where the emergency vehicle is located based on the image data; An avoidance strategy for the emergency vehicle is pushed according to the lane relationship, the driving direction and the relative position.

2. The method according to claim 1, wherein: The audio data includes audio collected by multiple audio collection devices arranged at different positions of the vehicle; Determining the direction of travel of the emergency vehicle and its relative position to the vehicle based on the collected audio data includes: extracting a sound frequency feature from the audio data, and determining whether an emergency vehicle exists based on the sound frequency feature; After determining that an emergency vehicle exists, the travel direction of the emergency vehicle and the relative position of the emergency vehicle to the vehicle are determined based on the sound frequency characteristics of the emergency vehicle in each audio frequency.

3. The method according to claim 2, wherein: The step of extracting a sound frequency feature from the audio data and determining whether an emergency vehicle exists based on the sound frequency feature comprises: For any audio in the audio data, perform noise filtering and automatic gain processing on the audio to obtain processed audio; Converting the processed audio from a time domain signal to a frequency domain signal to obtain a converted audio signal; Extracting frequency features of the converted audio signal to obtain the sound frequency features; It is determined whether an emergency vehicle exists based on the sound frequency characteristics and the frequency characteristics of a plurality of alarm sounds acquired in advance.

4. The method according to claim 2 or 3, wherein: Determining the direction of travel of the emergency vehicle and the relative position of the emergency vehicle to the vehicle according to the sound frequency characteristics of the emergency vehicle in each audio frequency includes: According to the position of each audio collection device in the vehicle and the sound frequency characteristics of the emergency vehicle in each audio, obtaining the time difference between different audio collection devices receiving the sound of the emergency vehicle and / or the intensity difference between different audio collection devices receiving the sound of the emergency vehicle at the same time; Determine the relative position of the emergency vehicle and the vehicle according to the time difference between different audio collection devices receiving the sound of the emergency vehicle and / or the intensity difference between different audio collection devices receiving the sound of the emergency vehicle at the same time; The driving direction of the emergency vehicle is determined according to the change of the sound intensity of the emergency vehicle collected by any audio collection device at adjacent moments, and the driving direction includes approaching the vehicle or moving away from the vehicle.

5. The method according to any one of claims 1 to 4, wherein: The step of determining whether to collect image data of the emergency vehicle according to the relative position includes: If the emergency vehicle is located behind the vehicle, it is determined that image data of the emergency vehicle needs to be collected; Otherwise, it is determined that it is not necessary to collect the image data of the emergency vehicle.

6. The method according to any one of claims 1 to 5, wherein: Determining the lane relationship between the lane where the vehicle is located and the lane where the emergency vehicle is located according to the image data includes: Extracting a red and blue light flickering area from the image data based on a deep learning algorithm, and obtaining a light source point of the red and blue light flickering area; Performing lane recognition on the image data to determine lane information; Determine the lane relationship according to the lane information and the light source points in the red and blue light areas, wherein the lane relationship is used to indicate whether the lane where the emergency vehicle is located is the same lane as the lane where the vehicle is located; Accordingly, the avoiding strategy for the emergency vehicle is pushed according to the lane relationship, the driving direction and the relative position, including: If the emergency vehicle and the own vehicle are traveling in the same lane, and the emergency vehicle is traveling towards the own vehicle, an avoidance strategy for the emergency vehicle is pushed according to the relative position.

7. The method according to claim 6, wherein: The pushing of the avoidance strategy for the emergency vehicle according to the relative position includes: Determining the distance between the emergency vehicle and the vehicle according to the relative position; If the distance is less than a preset distance threshold, an avoidance strategy is determined based on the road conditions on which the vehicle is traveling; the avoidance strategy includes a side-pull avoidance strategy or a lane-changing avoidance strategy; The avoidance strategy is pushed through at least one output interface of the vehicle terminal, the vehicle computer interface or the speaker.

8. The method according to claim 7, wherein: Determining the avoidance strategy based on the road conditions of the vehicle includes: If the road condition indicates that there are no other drivable roads on both sides of the vehicle, determining that the avoidance strategy is a side-by-side avoidance strategy; If the road condition indicates that there are other drivable roads on both sides of the vehicle, determining that the avoidance strategy is a lane change avoidance strategy; The pull-over avoidance strategy includes: activating the right turn signal, decelerating to a speed below a preset speed, and driving close to the edge of the right lane after deceleration; or activating the right turn signal, decelerating to a speed below a preset speed, and stopping close to the edge of the right lane after deceleration; The lane change avoidance strategy includes: changing the vehicle to another drivable lane.

9. The method according to claim 8, if it is determined that the avoidance strategy is a lane change avoidance strategy, the method further comprises: By detecting the image information or navigation information around the vehicle, the vehicle conditions in other drivable lanes are identified and the target lane is determined; The lane change avoidance strategy specifically includes: changing the vehicle lane to the target lane for driving.

10. The method according to any one of claims 1 to 9, further comprising: According to the avoidance strategy, the vehicle is controlled to avoid the emergency vehicle.

11. The method according to claim 3, further comprising: Determining the vehicle type of the emergency vehicle based on the sound frequency characteristics and the frequency characteristics of a plurality of alarm sounds acquired in advance; The vehicle types include: any one of an ambulance, a fire truck, a police car, and an engineering rescue vehicle; The avoidance strategy also includes the vehicle type of the emergency vehicle.

12. The method according to any one of claims 1 to 11, wherein: The acquiring of image data obtained by collecting images of the location of the emergency vehicle includes: The image acquisition device arranged at the rear of the vehicle is controlled to acquire an image of the rear of the vehicle to obtain the image data.

13. The method according to claim 3, wherein: The step of extracting the frequency feature of the converted audio signal to obtain the sound frequency feature includes: Performing peak detection on each signal segment in the converted audio signal, determining the frequency with the largest amplitude in each signal segment as the main frequency, and obtaining harmonics of the main frequency of each signal segment; The sound frequency characteristics are obtained according to the main frequency and harmonics of each signal segment.

14. The method according to claim 6, wherein: The extracting of the red and blue light flickering area from the image data based on the deep learning algorithm and obtaining the light source point of the red and blue light flickering area includes: Based on the YOLO detection algorithm, identifying the red and blue light flashing area from the image data; The brightness and color intensity of each pixel in the red and blue light flickering area are obtained, and based on the brightness and color intensity of each pixel, the light source point in the red and blue light flickering area is determined.

15. A multi-modal emergency vehicle processing device, comprising: A first processing unit, for determining a travel direction of the emergency vehicle and a relative position to the own vehicle based on the collected audio data; A second processing unit, configured to determine whether to collect image data of the emergency vehicle according to the relative position; A third processing unit is used to obtain image data obtained by collecting images of the position of the emergency vehicle when it is determined that image data needs to be collected; a fourth processing unit, configured to determine a lane relationship between a lane where the vehicle is located and a lane where the emergency vehicle is located according to the image data; The fifth processing unit is used to push an avoidance strategy for the emergency vehicle according to the lane relationship, the driving direction and the relative position.

16. The device according to claim 15, wherein: The audio data includes audio collected by a plurality of audio collection devices arranged at different positions of the vehicle, and the first processing unit includes: A first processing module, configured to extract a sound frequency feature from the audio data and determine whether an emergency vehicle exists based on the sound frequency feature; The second processing module is used to determine the travel direction of the emergency vehicle and the relative position of the emergency vehicle to the own vehicle according to the sound frequency characteristics of the emergency vehicle in each audio after determining the existence of the emergency vehicle.

17. The device according to claim 16, wherein: The first processing module comprises: A first processing submodule, configured to perform noise filtering and automatic gain processing on any audio in the audio data to obtain a processed audio; A second processing submodule, configured to convert the processed audio from a time domain signal into a frequency domain signal to obtain a converted audio signal; A third processing submodule extracts frequency features of the converted audio signal to obtain the sound frequency features; The fourth processing submodule determines whether there is an emergency vehicle based on the sound frequency characteristics and the frequency characteristics of multiple alarm sounds acquired in advance.

18. The device according to claim 16 or 17, wherein: The second processing module comprises: a fifth processing submodule, for obtaining, according to the position of each audio collection device in the vehicle and the sound frequency characteristics of the emergency vehicle in each audio, a time difference when different audio collection devices receive the sound of the emergency vehicle and / or an intensity difference when different audio collection devices receive the sound of the emergency vehicle at the same time; a sixth processing submodule, configured to determine the relative position of the emergency vehicle and the vehicle according to the time difference between different audio collection devices receiving the sound of the emergency vehicle and / or the intensity difference between different audio collection devices receiving the sound of the emergency vehicle at the same time; The seventh processing submodule is used to determine the driving direction of the emergency vehicle according to the change of the sound intensity of the emergency vehicle collected by any audio collection device at adjacent moments, and the driving direction includes approaching or moving away from the own vehicle.

19. The device according to any one of claims 15 to 18, wherein: The second processing unit comprises: A third processing module, configured to determine that image data of the emergency vehicle needs to be collected if the emergency vehicle is located behind the vehicle; Otherwise, it is determined that it is not necessary to collect the image data of the emergency vehicle.

20. The device according to any one of claims 15 to 19, wherein: The fourth processing unit comprises: A fourth processing module, configured to extract a red and blue light flickering area from the image data based on a deep learning algorithm, and obtain a light source point of the red and blue light flickering area; A fifth processing module, configured to perform lane recognition on the image data to determine lane information; a sixth processing module, configured to determine the lane relationship according to the lane information and the light source points in the red and blue light areas, wherein the lane relationship is used to indicate whether the lane where the emergency vehicle is located is the same lane as the lane where the vehicle is located; Accordingly, the fifth processing unit includes: The seventh processing module is used to push an avoidance strategy for the emergency vehicle according to the relative position if the lane in which the emergency vehicle and the own vehicle are traveling are the same lane and the driving direction of the emergency vehicle is approaching the own vehicle.

21. The device according to claim 20, wherein: The seventh processing module comprises: an eighth processing submodule, configured to determine the distance between the emergency vehicle and the vehicle according to the relative position; A ninth processing submodule, for determining an avoidance strategy based on the road conditions on which the vehicle is traveling if the distance is less than a preset distance threshold; the avoidance strategy includes a sideways avoidance strategy or a lane change avoidance strategy; The tenth processing submodule is used to push the avoidance strategy through at least one output interface of the vehicle terminal, the vehicle computer interface or the speaker.

22. The device according to claim 21, wherein The ninth processing submodule is specifically used for: If the road condition indicates that there are no other drivable roads on both sides of the vehicle, determining that the avoidance strategy is a side-by-side avoidance strategy; If the road condition indicates that there are other drivable roads on both sides of the vehicle, determining that the avoidance strategy is a lane change avoidance strategy; The pull-over avoidance strategy includes: activating the right turn signal, decelerating to a speed below a preset speed, and driving close to the edge of the right lane after deceleration; or activating the right turn signal, decelerating to a speed below a preset speed, and stopping close to the edge of the right lane after deceleration; The lane change avoidance strategy includes: changing the vehicle to another drivable lane.

23. The device according to claim 22, if it is determined that the avoidance strategy is a lane change avoidance strategy, the device further comprises: A sixth processing unit, configured to identify vehicles in other drivable lanes by detecting image information or navigation information around the vehicle, and determine the target lane; The lane change avoidance strategy specifically includes: changing the vehicle lane to the target lane for driving.

24. The device according to any one of claims 15 to 23, further comprising: The seventh processing unit is used to control the vehicle to avoid the emergency vehicle according to the avoidance strategy.

25. The apparatus according to claim 17, further comprising: an eighth processing unit, configured to determine the vehicle type of the emergency vehicle according to the sound frequency characteristics and the frequency characteristics of a plurality of alarm sounds acquired in advance; The vehicle types include: any one of an ambulance, a fire truck, a police car, and an engineering rescue vehicle; The avoidance strategy also includes the vehicle type of the emergency vehicle.

26. The device according to any one of claims 15 to 25, wherein: The third processing unit comprises: The eighth processing module is used to control the image acquisition device arranged at the rear of the vehicle to acquire the image data from the rear of the vehicle.

27. The device according to claim 17, wherein: The third processing submodule is specifically used for: Performing peak detection on each signal segment in the converted audio signal, determining the frequency with the largest amplitude in each signal segment as the main frequency, and obtaining harmonics of the main frequency of each signal segment; The sound frequency characteristics are obtained according to the main frequency and harmonics of each signal segment.

28. The device according to claim 20, wherein: The fourth processing module is specifically used for: Based on the YOLO detection algorithm, identifying the red and blue light flashing area from the image data; The brightness and color intensity of each pixel in the red and blue light flickering area are obtained, and based on the brightness and color intensity of each pixel, the light source point in the red and blue light flickering area is determined.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A vehicle comprising: A vehicle body, and a plurality of audio collection devices respectively arranged at different positions of the vehicle body; A processing device is arranged in the vehicle body, and the processing device is used to execute the multimodal-based emergency vehicle processing method described in any one of claims 1 to 14; the processing device includes an audio processing unit and a vehicle control unit.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

32. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

Images