Vehicle avoidance warning method, program product, electronic equipment and storage medium

Abstract

The invention provides a vehicle avoidance warning method, a program product, electronic equipment and a storage medium. The method comprises the following steps: receiving vehicle external audio data and vehicle external image data collected in a vehicle driving process; performing feature comparison on the vehicle external audio data and preset vehicle audio features to obtain a feature comparison result; under the condition that the feature comparison result meets the condition, detecting the external image data of the vehicle to obtain a detection result; and if the detection result shows that the avoidance object exists, performing avoidance warning, so that a driver in the vehicle controls the vehicle to avoid based on the avoidance warning. Whether the avoidance object exists or not is determined through the vehicle external audio data and the vehicle external image data, the detection accuracy is improved, the situation of false detection caused by single data detection is improved, efficient cooperation of vehicles is achieved, and the driving safety and the traffic efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of traffic technology, and in particular to a vehicle avoidance warning method, a program product, an electronic device and a storage medium. Background Art

[0002] In this era of rapid technological development, cars are not only becoming more and more intelligent, but also have better sound insulation. Admittedly, this has greatly improved the comfort of the users in the car, but it is actually a double-edged sword. In some situations, "good sound insulation" becomes an obstacle to civilized driving. For example, when encountering some vehicles that need to be avoided urgently during driving, the driver cannot find and take evasive measures in time. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a vehicle avoidance warning method, program product, electronic device and storage medium, which obtain external audio data and external image data of the vehicle, detect whether there is an avoidance object through multimodal data, and enable the driver in the vehicle to control the vehicle to avoid based on the avoidance warning, thereby achieving efficient vehicle collaboration and improving driving safety and traffic efficiency.

[0004] In a first aspect, an embodiment of the present application provides a vehicle avoidance warning method, comprising: receiving vehicle external audio data and vehicle external image data collected during vehicle driving; performing feature comparison on the vehicle external audio data and preset vehicle audio features to obtain feature comparison results; when the feature comparison results meet conditions, performing detection on the vehicle external image data to obtain detection results; if the detection results indicate that there is an avoidance object, performing an avoidance warning, so that the driver in the vehicle controls the vehicle to avoid based on the avoidance warning.

[0005] In the above implementation process, the vehicle's external audio data and vehicle's external image data are obtained, and feature comparison is first performed based on the vehicle's external audio data. When the feature comparison result meets the conditions, the vehicle's external image data is detected, and there is no need to detect each vehicle's external image data, thereby improving computing efficiency and reducing resource consumption. In addition, the vehicle's external audio data and vehicle's external image data are used to jointly determine whether there is an avoidance object, thereby improving the accuracy of detection, improving the situation of false detection in single data detection, achieving efficient vehicle collaboration, and improving driving safety and traffic efficiency.

[0006] Optionally, in an embodiment of the present application, the preset vehicle audio feature includes a reference spectrum graph of the preset vehicle audio; the feature comparison result includes a first feature comparison result; the vehicle external audio data is feature compared with the preset vehicle audio feature to obtain a feature comparison result, including: sampling the sound signal in the vehicle external audio data at a preset sampling rate to obtain a discrete time signal; converting the discrete time signal from the time domain to the frequency domain to obtain a spectrum graph; calculating the similarity between the spectrum graph and the reference spectrum graph of the preset vehicle audio to obtain a first feature comparison result; the first feature comparison result is used to characterize the spectral similarity between the vehicle external audio data and the preset vehicle audio.

[0007] In the above implementation process, by comparing the spectrum of the vehicle's external audio and the preset vehicle audio, the subtle differences in the audio signal are captured, thereby improving the accuracy of recognition. Spectral analysis has a certain robustness to noise and interference in the audio signal. Through appropriate preprocessing and feature extraction, the impact of environmental noise on the recognition results can be reduced to a certain extent.

[0008] Optionally, in an embodiment of the present application, the preset vehicle audio feature includes a reference energy value of the preset vehicle audio; the feature comparison result includes a second feature comparison result; the vehicle external audio data is feature compared with the preset vehicle audio feature to obtain a feature comparison result, including: calculating the energy value of the sound signal in the vehicle external audio data; performing energy similarity calculation on the energy value and the reference energy value of the preset vehicle audio to obtain a second feature comparison result; the second feature comparison result is used to characterize the energy similarity between the vehicle external audio data and the preset vehicle audio.

[0009] In the above implementation process, the energy similarity between the external audio data of the vehicle and the preset vehicle audio is calculated, and the degree of proximity of the two audio signals in overall strength can be intuitively understood. The calculation method is simple, and a fast and effective preliminary screening method can be used, which can achieve fast response in real-time audio processing.

[0010] Optionally, in an embodiment of the present application, the feature comparison result includes spectral similarity and energy similarity; the feature comparison result satisfies a condition, including: obtaining the spectral similarity and energy similarity obtained for the vehicle external audio data for the same preset vehicle audio, if the spectral similarity is within a first preset range and the energy similarity is within a second preset range, then the feature comparison result satisfies the condition.

[0011] In the above implementation process, spectrum similarity and energy similarity each have their own advantages in determining whether the vehicle external audio data is the preset vehicle audio, and the two are combined to improve the accuracy and efficiency of recognition.

[0012] Optionally, in an embodiment of the present application, if the detection result characterizes an avoidance object, the detection result includes position information of the avoidance object relative to the vehicle; when the feature comparison result satisfies the conditions, the vehicle external image data is detected to obtain the detection result, including: obtaining a reference image corresponding to the feature comparison result; the reference image is a pre-collected image containing a vehicle to be avoided; similarity is calculated between the vehicle external image data and the reference image to obtain an image similarity result; when the image similarity result is greater than a preset threshold, target detection is performed on the vehicle external image data to obtain the avoidance object in the vehicle external image data and the image coordinates of the avoidance object; the avoidance object is the avoidance object in the vehicle external image data; the image coordinates of the avoidance object are converted to obtain position information of the avoidance object relative to the vehicle, and the position information includes distance and / or direction angle.

[0013] In the above implementation process, by calculating the similarity between the vehicle's external image data and the reference image, a preliminary judgment is made as to whether there is an avoidance object, and target detection is performed on the vehicle's external image data to improve the accuracy of avoidance object detection, so that timely response can be made and correct obstacle avoidance decisions can be made during driving.

[0014] Optionally, in an embodiment of the present application, the method also includes: receiving a vehicle avoidance request sent by the avoidance object, the vehicle avoidance request including driving status information of the avoidance object; the driving status information includes at least one of position information, driving speed, driving direction, speed change information and vehicle lights; dynamically adjusting the driving route based on the driving status information, so that the driver in the vehicle avoids the avoidance object based on the adjusted route.

[0015] In the above implementation process, the avoidance object can send a vehicle avoidance request to the vehicle, and the vehicle dynamically adjusts the driving route based on the driving status information to safely and promptly avoid emergency vehicles or other vehicles that need to be avoided. This not only improves road safety, but also helps to improve overall traffic efficiency.

[0016] Optionally, in an embodiment of the present application, receiving vehicle external audio data and vehicle external image data collected during the vehicle's driving process includes: collecting the vehicle's initial audio data during the vehicle's driving process through a microphone arranged outside the vehicle; performing sound signal preprocessing on the vehicle's initial audio data to obtain vehicle external audio data; the sound signal preprocessing includes equalization and / or denoising; acquiring vehicle external image data through the vehicle's panoramic imaging system; the vehicle external image data includes images of at least one orientation.

[0017] In the above implementation process, the initial audio data of the vehicle during driving is collected by a microphone installed outside the vehicle, and the sound can be accurately captured in various environments. The sound signal is preprocessed through the initial audio data of the vehicle to improve the quality of the sound signal, laying a good foundation for subsequent sound analysis and feature comparison. In addition, the vehicle's external image data is obtained through the vehicle's panoramic imaging system, without the need to set up additional cameras, saving resources and costs.

[0018] On the second aspect, the embodiment of the present application also provides a vehicle avoidance warning device, including: a data receiving module, used to receive vehicle external audio data and vehicle external image data collected during the vehicle's driving process; an audio comparison module, used to perform feature comparison on the vehicle's external audio data with preset vehicle audio features to obtain feature comparison results; an image detection module, used to detect the vehicle's external image data and obtain a detection result when the feature comparison result meets the conditions; a warning module, used to issue an avoidance warning if the detection result indicates that there is an avoidance object, so that the driver in the vehicle can control the vehicle to avoid based on the avoidance warning.

[0019] In a third aspect, an embodiment of the present application further provides a computer program product, including computer program instructions, which, when executed by a processor, execute the method provided by the first aspect or any one of the implementations of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, the memory storing computer program instructions, and the computer program instructions, when executed by the processor, execute the method provided by the first aspect or any one of the implementations of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method provided by the first aspect or any one of the implementations of the first aspect is executed.

[0022] The vehicle avoidance warning method, program product, electronic device and storage medium provided by the present application are used to obtain the vehicle external audio data and vehicle external image data, first perform feature comparison based on the vehicle external audio data, and detect the vehicle external image data when the feature comparison result meets the conditions, without the need to detect each vehicle external image data, thereby improving computing efficiency and reducing resource consumption. In addition, the vehicle external audio data and vehicle external image data are used to jointly determine whether there is an avoidance object, thereby improving the accuracy of detection, improving the situation of false detection in single data detection, achieving efficient vehicle collaboration, and improving driving safety and traffic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flow chart of a vehicle avoidance warning method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a vehicle avoidance warning device provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following embodiments of the technical solution of the present application will be described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0029] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0030] When driving, you may encounter some vehicles that need to be avoided urgently, such as police cars, ambulances, fire trucks, or vehicles that are reversing. In order to improve traffic safety and efficiency, the driver needs to promptly detect these situations that need to be avoided and perform avoidance operations. However, due to the relatively good sound insulation effect of current vehicles and the high concentration of the driver during driving, it is difficult to notice the sound of these vehicles. In addition, due to the limited field of vision, these vehicles cannot be found in time, so it is impossible to control the vehicle to give way in time, which brings some inconvenience.

[0031] A vehicle avoidance warning method provided in an embodiment of the present application obtains external audio data and external image data of the vehicle, detects whether there is an avoidance object through multimodal data, and enables the driver in the vehicle to control the vehicle to avoid based on the avoidance warning, thereby achieving efficient vehicle collaboration and improving driving safety and traffic efficiency.

[0032] See also Figure 1 The flowchart of a vehicle avoidance warning method provided by an embodiment of the present application is shown. The vehicle avoidance warning method provided by an embodiment of the present application can be applied to an electronic device, which can be deployed inside a vehicle. The electronic device can include a physical device such as a server, a PC, a tablet computer, or an in-vehicle smart device, or can also be a virtual device such as a virtual machine or a container. The vehicle avoidance warning method may include:

[0033] Step S110: receiving vehicle external audio data and vehicle external image data collected during vehicle driving.

[0034] Step S120: performing feature comparison on the vehicle external audio data and the preset vehicle audio features to obtain feature comparison results.

[0035] Step S130: When the feature comparison result satisfies the conditions, the vehicle external image data is detected to obtain the detection result.

[0036] Step S140: If the detection result indicates that there is an avoidance object, an avoidance warning is issued, so that the driver in the vehicle controls the vehicle to avoid the object based on the avoidance warning.

[0037] In step S110, the vehicle external audio data refers to the audio data collected by the audio collection device installed outside the vehicle. The vehicle external image data refers to the image data collected by the cameras installed at different positions outside the vehicle. The vehicle external image data may also refer to the video frames in the video data collected by the vehicle external camera. The vehicle external camera may be configured separately, or the camera in the vehicle's original AVM (Around View Monitor) may be used.

[0038] In step S120, the preset vehicle audio feature is obtained by extracting the features of the preset vehicle audio. The preset vehicle audio may include one or more, and the preset vehicle audio may be the sound carried or emitted by the vehicle to be avoided, such as the sirens or engine sounds of police cars, ambulances or fire trucks, the sirens of other vehicles, or the warning sound of a vehicle reversing, etc.

[0039] These preset vehicle audios can be collected in advance or obtained through the network, and the preset vehicle audios are feature extracted to obtain preset vehicle audio features, and the preset vehicle audio features include at least one of spectral features, energy features, time domain features and statistical features. Among them, spectral features refer to the performance of audio signals in the frequency domain, and spectral features include the frequency components with the highest energy in the audio signal or the distribution of different frequency components in the audio signal. Energy features refer to the energy distribution of audio signals in the time domain or frequency domain, including short-time energy, total energy or energy envelope. Time domain features refer to the performance of audio signals in the time domain, that is, the changes of signals over time. Time domain features include time domain waveforms or zero crossing rates. Statistical features refer to representative and distinguishable feature parameters extracted from audio signals by statistical methods. Statistical features include the mean, variance, standard deviation or skewness of audio signals. These preset vehicle audio features can be stored in a database.

[0040] The received vehicle external audio data is also subjected to feature extraction, and feature extraction may be performed on at least one of spectrum features, energy features, time domain features, and statistical features. The vehicle external audio data is then feature-compared with the preset vehicle audio features to obtain feature comparison results.

[0041] It is understandable that if the preset vehicle audio features are spectral features and energy features, then the vehicle external audio data should also extract spectral features and energy features so that they correspond in the subsequent feature comparison. The spectral features of the vehicle external audio data are compared with the spectral features in the preset vehicle audio features to obtain the feature comparison results of the spectral features. The energy features of the vehicle external audio data are compared with the energy features in the preset vehicle audio features to obtain the feature comparison results of the energy features. The feature comparison results may include the spectral similarity of the spectral features and the energy similarity of the energy features.

[0042] In step S130, after obtaining the feature comparison result, it is determined whether the feature comparison result meets the condition. If the feature comparison result meets the condition, the vehicle external image data is detected to obtain the detection result. If the feature comparison result does not meet the condition, the vehicle external image data may not be detected, which can reduce the amount of image detection calculation and reduce resource consumption.

[0043] The detection method of the vehicle external image data may include using a pre-trained neural network model for detection, or using an image similarity algorithm for detection. The following describes the two methods respectively.

[0044] For detection using a neural network model: First, a large number of sample images of the vehicle's exterior need to be collected, including vehicles or objects that need to be avoided. These sample images can come from various sources, such as traffic monitoring cameras, vehicle-mounted cameras, mobile phone shots, etc. The collected sample images of the vehicle's exterior need to be annotated, and you can manually or automatically mark whether the image contains the target vehicle, as well as the location of the vehicle (which can be represented by a bounding box). Before training the model, the image data also needs to be preprocessed, including resizing the image, normalizing pixel values, and data augmentation (such as rotation, scaling, flipping, etc.) to increase the generalization ability of the model.

[0045] Select the appropriate neural network model according to the specific task requirements. For the avoidance object (vehicle) detection task, you can use convolutional neural network (CNN) and its variants, such as Faster R-CNN, YOLO, SSD (Single Shot MultiBoxDetector), etc. Use the labeled sample image set to train the selected neural network model. During the training process, the model will learn how to extract features from the input image and determine whether the image contains the target vehicle and the location of the vehicle based on these features. The target vehicle is the avoidance object.

[0046] After the model is trained, the collected vehicle external image data is input into the trained model for detection. If a target vehicle is detected, the model will output the location information of the target vehicle (such as bounding box coordinates), that is, the detection result includes whether there is an avoidance object and / or the location of the avoidance object.

[0047] For detection using image similarity algorithms: the vehicle external image data is similarity calculated with the reference image to obtain the image similarity result. The reference image here is a pre-collected image of the vehicle that needs to be avoided. Among them, the similarity calculation methods include feature-based algorithms and pixel-based algorithms. The feature-based algorithms include SIFT (Scale Invariant Feature Transform) or SURF (Speeded Robust Features), and the pixel-based algorithms include histogram comparison or template matching.

[0048] If the image similarity result is greater than a preset threshold, the detection result represents that there is an avoidance object in the vehicle external image data; if the image similarity result is not greater than the preset threshold, the detection result represents that there is no avoidance object in the vehicle external image data.

[0049] In step S140, if the detection result indicates that there is an avoidance object, an avoidance warning is issued so that the driver in the vehicle controls the vehicle to avoid the object based on the avoidance warning. For example, the position of the vehicle to be avoided or the prompt text "There is an avoidance vehicle directly behind, the distance is 10 meters, please avoid it" is displayed on the screen inside the vehicle, and a voice prompt or warning sound can also be issued through the speaker.

[0050] In the above implementation process, the vehicle's external audio data and vehicle's external image data are obtained, and feature comparison is first performed based on the vehicle's external audio data. When the feature comparison result meets the conditions, the vehicle's external image data is detected, and there is no need to detect each vehicle's external image data, thereby improving computing efficiency and reducing resource consumption. In addition, the vehicle's external audio data and vehicle's external image data are used to jointly determine whether there is an avoidance object, thereby improving the accuracy of detection, improving the situation of false detection in single data detection, achieving efficient vehicle collaboration, and improving driving safety and traffic efficiency.

[0051] Optionally, in an embodiment of the present application, the preset vehicle audio feature includes a reference spectrum of the preset vehicle audio; the reference spectrum refers to the spectrum feature of the preset vehicle audio, and the distribution of different frequency components in the audio signal in the reference spectrum. The feature comparison result includes a first feature comparison result; the feature comparison result is obtained by performing a feature comparison between the vehicle external audio data and the preset vehicle audio feature, including:

[0052] The sound signal in the vehicle's external audio data is sampled at a preset sampling rate to obtain a discrete time signal. For example, according to the Nyquist sampling theorem, the sound signal is sampled at a sampling rate twice the highest frequency of the signal to convert the continuous time signal into a discrete time signal. Each sample value obtained by sampling can also be mapped to the closest quantization level to form a discrete value signal for easy digital processing.

[0053] The discrete time signal is converted from the time domain to the frequency domain to obtain a spectrum diagram. For example, the processed discrete time signal is transformed by FFT (Fourier transform) to convert the signal from the time domain to the frequency domain to obtain a spectrum diagram.

[0054] The similarity between the spectrum graph and the reference spectrum graph of the preset vehicle audio is calculated to obtain a first feature comparison result; the first feature comparison result is used to characterize the spectrum similarity between the vehicle external audio data and the preset vehicle audio. For example, a similarity measurement method such as Euclidean distance or cosine similarity can be used to compare the spectrum graph and the reference spectrum graph of the preset vehicle audio to obtain spectrum similarity. The spectrum similarity is a numerical value that reflects the degree of similarity between the two in terms of spectrum features (for example, complete inconsistency is 0%, and complete consistency is 100%).

[0055] In the implementation process of the above embodiment: by comparing the spectrum of the vehicle's external audio and the preset vehicle audio, the subtle differences in the audio signal are captured, thereby improving the accuracy of recognition. Spectral analysis has a certain robustness to noise and interference in the audio signal. Through appropriate preprocessing and feature extraction, the impact of environmental noise on the recognition result can be reduced to a certain extent.

[0056] Optionally, in an embodiment of the present application, the preset vehicle audio feature includes a reference energy value of the preset vehicle audio; the reference energy value is obtained by extracting the energy feature of the preset vehicle audio, and the calculation method is, for example, calculating the sum of the squares of the audio signals of the preset vehicle audio divided by the number of samples to obtain an energy average value, and using the energy average value as the reference energy value. The feature comparison result includes a second feature comparison result; the feature comparison result obtained by performing feature comparison between the vehicle external audio data and the preset vehicle audio feature includes:

[0057] The energy value of the sound signal in the vehicle external audio data is calculated, for example, the sound signal can be accumulated to obtain its total energy value, and the total energy value is divided by the sampling point to obtain the energy value. (The energy value is the energy average value).

[0058] The energy value and the reference energy value of the preset vehicle audio are used to calculate the energy similarity to obtain the second feature comparison result; the second feature comparison result is used to characterize the energy similarity between the vehicle external audio data and the preset vehicle audio. For example, the formula can be used: Wherein, B is the second feature comparison result, E1 is the energy value, and E2 is the reference energy value of the preset vehicle audio. Of course, other algorithms can also be used to calculate the second feature comparison result, such as calculating the energy difference between the two (the energy value and the reference energy value of the preset vehicle audio), and normalizing the difference, for example, between 0 and 1 (or between 0% and 100%).

[0059] In the implementation process of the above embodiment: the energy similarity between the external audio data of the vehicle and the preset vehicle audio is calculated, and the degree of proximity of the two audio signals in overall strength is intuitively understood. The calculation method is simple, and a fast and effective preliminary screening method can be used, which can achieve fast response in real-time audio processing.

[0060] Optionally, in an embodiment of the present application, the feature comparison result includes spectral similarity and energy similarity; the feature comparison result satisfies a condition, including: obtaining the spectral similarity and energy similarity of the vehicle external audio data for the same preset vehicle audio; if the spectral similarity is within a first preset range and the energy similarity is within a second preset range, then the feature comparison result satisfies the condition.

[0061] For example, if the preset vehicle audio is an ambulance siren, the spectrum similarity and energy similarity in the feature comparison result should be obtained by comparing the vehicle external audio data with the ambulance siren. That is, the spectrum similarity and energy similarity should be obtained by comparing the vehicle external audio data with the same preset vehicle audio.

[0062] If the spectral similarity is within the first preset range and the energy similarity is within the second preset range, it means that both the spectral similarity and the energy similarity are characterized by high similarity, and it can be confirmed that the feature comparison result meets the conditions. The first preset range and the second preset range can be set according to actual needs, and the embodiment of the present application does not limit this.

[0063] In the implementation process of the above embodiment: spectrum similarity and energy similarity each have their own advantages in determining whether the vehicle external audio data is the preset vehicle audio, and the two are combined to improve the accuracy and efficiency of recognition.

[0064] Optionally, in an embodiment of the present application, the detection result may include whether there is an avoidance object. If the detection result indicates that there is an avoidance object, the detection result includes position information of the avoidance object relative to the vehicle. If a neural network model is used to obtain the detection result, the model will output the position information of the avoidance object (such as bounding box coordinates), and convert the bounding box coordinates into position information of the avoidance object relative to the vehicle, so that the driver can more intuitively know the position of the avoidance object and improve the accuracy of controlling the vehicle to avoid.

[0065] When the feature comparison result meets the conditions, the vehicle external image data is detected to obtain the detection results, including:

[0066] Obtain a reference image corresponding to the feature comparison result; the reference image is a pre-collected image containing the vehicle to be avoided. The reference image corresponding to the feature comparison result means that if the feature comparison result is compared based on the sound of an ambulance siren, then the reference image corresponding to the feature comparison result is the image of the ambulance. That is, the comparison object of the feature comparison result and the object in the reference image are the same.

[0067] The vehicle external image data and the reference image are subjected to similarity calculation to obtain an image similarity result. The similarity calculation method is as described above.

[0068] When the image similarity result is greater than a preset threshold, target detection is performed on the vehicle external image data to obtain the avoidance object in the vehicle external image data and the image coordinates of the avoidance object; the avoidance object is the avoidance object in the vehicle external image data.

[0069] The following describes the implementation process of the method for object detection on the external image data of the vehicle. First, build a target detection model. You can use the on-board camera to capture the image data of the outside of the vehicle and use these image data as sample data. Select a target detection algorithm suitable for the external image data of the vehicle. For example, it can be an algorithm based on deep learning (such as YOLO, SSD, Faster R-CNN, etc.) or a traditional image processing method (such as edge detection, template matching, etc.). Annotate the avoidance objects (such as vehicles, pedestrians, obstacles, etc.) in the sample data to obtain an annotated image dataset. The annotation can include information such as the category or bounding box of the avoidance object. Use the annotated dataset to train the selected target detection model. During the training process, the model will learn how to identify the avoidance objects in the image and predict their bounding boxes.

[0070] In one embodiment, the trained object detection model can be deployed on a vehicle-mounted computer or server to process vehicle external image data in real time.

[0071] After that, the vehicle external image data is input into the trained target detection model for target detection, and the target detection model outputs the avoidance object and the bounding box of the avoidance object in the vehicle external image data. According to the bounding box of the avoidance object predicted by the model, the image coordinates of the avoidance object in the image can be extracted. This usually includes the coordinates of the upper left corner and lower right corner of the bounding box (or the coordinates of the center point and the width and height information). The image coordinates are used to represent the position and size of the avoidance object in the image.

[0072] The image coordinates of the avoidance object are converted to obtain the position information of the avoidance object relative to the vehicle, which includes the distance and / or direction angle. The image coordinates are converted to the actual world coordinates (for example, for path planning and obstacle avoidance decisions). This process requires camera calibration and coordinate conversion calculation.

[0073] In an optional embodiment, based on the extracted image coordinates of the avoidance object and other sensor information (such as radar, laser radar, etc.), the vehicle's intelligent driving system can automatically make obstacle avoidance decisions and control the vehicle to drive safely. Further, after knowing the distance and / or direction angle of the avoidance object, the driver can be reminded that there is an avoidance vehicle directly behind, the distance is 10 meters, and please pay attention to avoid it.

[0074] In the implementation process of the above embodiment: by calculating the similarity between the vehicle's external image data and the reference image, a preliminary judgment is made as to whether there is an avoidance object, and target detection is performed on the vehicle's external image data to improve the accuracy of avoidance object detection, so that timely response can be made and correct obstacle avoidance decisions can be made during driving.

[0075] Optionally, in an embodiment of the present application, the method further includes: receiving a vehicle avoidance request sent by an avoidance object, the vehicle avoidance request including driving status information of the avoidance object; the driving status information includes at least one of position information, driving speed, driving direction, speed change information, and vehicle lights. For example, vehicle communication technology (V2X, including V2V and V2I) can be used to share vehicle driving status information. For example, when an ambulance (avoidance object) is driving, other vehicles need to avoid it as much as possible. Therefore, a vehicle avoidance request can be sent to vehicles within a preset range through vehicle-to-vehicle communication. The vehicle avoidance request includes driving status information. After receiving the vehicle avoidance request, other vehicles can process it in a timely manner to play a role in avoidance.

[0076] In an optional embodiment, the received vehicle avoidance request may be analyzed, for example, the form speed and route may be analyzed to identify whether it is issued by a vehicle that needs to be avoided. If it is issued by a vehicle to be avoided, avoidance is performed.

[0077] Dynamically adjust the driving route based on driving status information so that the driver in the vehicle can avoid the avoidance object based on the adjusted route. For example, the driving route and speed of the avoidance object are used to predict its possible driving path. Generate an avoidance strategy based on the vehicle's own driving conditions. The avoidance strategy may include different actions such as slowing down, changing lanes, stopping, or dynamically replanning the driving route. The on-board system notifies the driver of the avoidance strategy and recommended driving route through the instrument panel, head-up display (HUD) or voice prompts. During the avoidance process, the vehicle continuously monitors the state changes of the avoidance object and the surrounding environment.

[0078] In the implementation process of the above embodiment: the avoidance object can send a vehicle avoidance request to the vehicle, and the vehicle dynamically adjusts the driving route based on the driving status information to safely and timely avoid emergency vehicles or other vehicles that need to be avoided. This not only improves road safety, but also helps to improve overall traffic efficiency.

[0079] Optionally, in the embodiment of the present application, receiving the vehicle external audio data and the vehicle external image data collected during the vehicle driving process includes:

[0080] The initial audio data of the vehicle during the driving process is collected by a microphone arranged outside the vehicle. In an optional embodiment, the microphone outside the vehicle can be installed at the rear of the vehicle, and one or more microphones can be installed. A microphone with high sensitivity and high signal-to-noise ratio can be selected to ensure that the sound can be accurately captured in various environments. It is also possible to select a suitable frequency response range according to the type of sound to be collected (such as vehicle horns, engine sounds, etc.).

[0081] Perform sound signal preprocessing on the initial audio data of the vehicle to obtain the external audio data of the vehicle; the sound signal preprocessing includes equalization and / or denoising. Equalization refers to adjusting the frequency response of the audio signal to compensate for the frequency distortion in the pickup or transmission process, or to strengthen or suppress certain frequency components according to application requirements. Denoising refers to removing unwanted background noise from the audio signal. Equalization methods include achieving equalization by adjusting the gain of different frequency bands, or focusing on gain adjustment of specific frequencies. Denoising methods include spectral subtraction, Wiener filtering, or adaptive filtering.

[0082] The vehicle's external image data is obtained through the vehicle's panoramic imaging system; the vehicle's external image data includes images of at least one orientation. The vehicle's panoramic imaging system includes cameras installed in at least one orientation, and each camera of the vehicle's panoramic imaging system captures the vehicle's external image data of each orientation in real time. Afterwards, the image processing algorithm can be used to correct the distortion of the images captured by each camera, because the camera lens usually produces distortion.

[0083] In the implementation process of the above embodiment: the initial audio data of the vehicle during driving is collected by a microphone installed outside the vehicle, so that the sound can be accurately captured in various environments, and the sound signal is preprocessed through the initial audio data of the vehicle to improve the quality of the sound signal, laying a good foundation for subsequent sound analysis and feature comparison. And the external image data of the vehicle is obtained through the panoramic imaging system of the vehicle, without the need to set up additional cameras, saving resources and costs.

[0084] See also Figure 2 The schematic diagram of the structure of the vehicle avoidance warning device provided in the embodiment of the present application is shown; the embodiment of the present application provides a vehicle avoidance warning device 200, including:

[0085] The data receiving module 210 is used to receive the vehicle external audio data and vehicle external image data collected during the vehicle driving process;

[0086] The audio comparison module 220 is used to compare the vehicle external audio data with the preset vehicle audio features to obtain a feature comparison result;

[0087] The image detection module 230 is used to detect the vehicle external image data and obtain the detection result when the feature comparison result meets the conditions;

[0088] The warning module 240 is configured to issue an avoidance warning if the detection result indicates that there is an avoidance object, so that the driver in the vehicle controls the vehicle to avoid the object based on the avoidance warning.

[0089] Optionally, in an embodiment of the present application, the vehicle avoidance warning device 200, the preset vehicle audio features include a reference spectrum diagram of the preset vehicle audio; the feature comparison result includes a first feature comparison result; the audio comparison module 220 is specifically used to sample the sound signal in the vehicle external audio data according to a preset sampling rate to obtain a discrete time signal; convert the discrete time signal from the time domain to the frequency domain to obtain a spectrum diagram; calculate the similarity between the spectrum diagram and the reference spectrum diagram of the preset vehicle audio to obtain a first feature comparison result; the first feature comparison result is used to characterize the spectral similarity between the vehicle external audio data and the preset vehicle audio.

[0090] Optionally, in an embodiment of the present application, the vehicle avoidance warning device 200, the preset vehicle audio feature includes a reference energy value of the preset vehicle audio; the feature comparison result includes a second feature comparison result; the audio comparison module 220 is specifically used to calculate the energy value of the sound signal in the vehicle external audio data; the energy value and the reference energy value of the preset vehicle audio are calculated to obtain the second feature comparison result; the second feature comparison result is used to characterize the energy similarity between the vehicle external audio data and the preset vehicle audio.

[0091] Optionally, in an embodiment of the present application, the vehicle avoidance warning device 200, the feature comparison result includes spectral similarity and energy similarity; the feature comparison result satisfies a condition, including: obtaining the spectral similarity and energy similarity obtained for the vehicle external audio data for the same preset vehicle audio, if the spectral similarity is within a first preset range, and the energy similarity is within a second preset range, then the feature comparison result satisfies the condition.

[0092] Optionally, in an embodiment of the present application, the vehicle avoidance warning device 200, if the detection result indicates that there is an avoidance object, the detection result includes the position information of the avoidance object relative to the vehicle; the image detection module 230 is used to obtain a reference image corresponding to the feature comparison result; the reference image is a pre-collected image containing a vehicle to be avoided; the vehicle external image data and the reference image are subjected to similarity calculation to obtain an image similarity result; when the image similarity result is greater than a preset threshold, target detection is performed on the vehicle external image data to obtain the avoidance object in the vehicle external image data and the image coordinates of the avoidance object; the avoidance object is the avoidance object in the vehicle external image data; the image coordinates of the avoidance object are converted to obtain the position information of the avoidance object relative to the vehicle, and the position information includes distance and / or direction angle.

[0093] Optionally, in an embodiment of the present application, the vehicle avoidance warning device 200 further includes a vehicle communication and route planning module, which is used to receive a vehicle avoidance request sent by the avoidance object, the vehicle avoidance request including driving status information of the avoidance object; the driving status information includes at least one of position information, driving speed, driving direction, speed change information and vehicle lights; the driving route is dynamically adjusted based on the driving status information, so that the driver in the vehicle avoids the avoidance object based on the adjusted route.

[0094] Optionally, in an embodiment of the present application, the vehicle avoidance warning device 200, the data receiving module 210, is specifically used to collect the vehicle's initial audio data during the vehicle's driving process through a microphone arranged outside the vehicle; perform sound signal preprocessing on the vehicle's initial audio data to obtain vehicle external audio data; the sound signal preprocessing includes equalization and / or denoising processing; obtain vehicle external image data through the vehicle's panoramic imaging system; the vehicle external image data includes images of at least one orientation.

[0095] It should be understood that the device corresponds to the above-mentioned vehicle avoidance warning method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0096] See also Figure 3 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0097] Figure 3 Each component shown in can be implemented by hardware, software or a combination thereof. The electronic device 300 may be a physical device, such as a server, a PC, etc., or a virtual device, such as a virtual machine, a virtualized container, etc. Moreover, the electronic device 300 is not limited to a single device, but may also be a combination of multiple devices or a cluster consisting of a large number of devices.

[0098] An embodiment of the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is executed.

[0099] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0100] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0102] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A vehicle avoidance warning method, characterized in that: include: Receiving vehicle external audio data and vehicle external image data collected during vehicle driving; Performing feature comparison on the vehicle external audio data and preset vehicle audio features to obtain a feature comparison result; When the feature comparison result satisfies a condition, detecting the vehicle external image data to obtain a detection result; If the detection result indicates that there is an avoidance object, an avoidance warning is issued, so that the driver in the vehicle controls the vehicle to avoid the object based on the avoidance warning.

2. The method according to claim 1, characterized in that The preset vehicle audio feature includes a reference frequency spectrum of the preset vehicle audio; the feature comparison result includes a first feature comparison result; and performing feature comparison between the vehicle external audio data and the preset vehicle audio feature to obtain the feature comparison result includes: Sampling the sound signal in the vehicle external audio data at a preset sampling rate to obtain a discrete time signal; Convert the discrete time signal from the time domain to the frequency domain to obtain a frequency spectrum; The similarity between the spectrum graph and the reference spectrum graph of the preset vehicle audio is calculated to obtain the first feature comparison result; the first feature comparison result is used to characterize the spectrum similarity between the vehicle external audio data and the preset vehicle audio.

3. The method according to claim 1, characterized in that The preset vehicle audio feature includes a reference energy value of the preset vehicle audio; the feature comparison result includes a second feature comparison result; and performing feature comparison between the vehicle external audio data and the preset vehicle audio feature to obtain the feature comparison result includes: Calculating the energy value of the sound signal in the vehicle external audio data; The energy value and the reference energy value of the preset vehicle audio are subjected to energy similarity calculation to obtain the second feature comparison result; the second feature comparison result is used to characterize the energy similarity between the vehicle external audio data and the preset vehicle audio.

4. The method according to claim 1, characterized in that The feature comparison result includes spectrum similarity and energy similarity; the feature comparison result satisfies the following conditions: The spectrum similarity and the energy similarity obtained for the vehicle external audio data for the same preset vehicle audio are obtained. If the spectrum similarity is within a first preset range and the energy similarity is within a second preset range, it indicates that the feature comparison result meets the condition.

5. The method according to claim 1, characterized in that If the detection result indicates that there is an avoidance object, the detection result includes position information of the avoidance object relative to the vehicle; When the feature comparison result satisfies the condition, the vehicle external image data is detected to obtain the detection result, including: Obtaining a reference image corresponding to the feature comparison result; the reference image is a pre-collected image containing the vehicle to be avoided; Calculating the similarity between the vehicle external image data and the reference image to obtain an image similarity result; When the image similarity result is greater than a preset threshold, target detection is performed on the vehicle external image data to obtain an avoidance object in the vehicle external image data and image coordinates of the avoidance object; the avoidance object is the avoidance object in the vehicle external image data; The image coordinates of the avoidance object are converted to obtain position information of the avoidance object relative to the vehicle, where the position information includes distance and / or direction angle.

6. The method according to claim 1, characterized in that The method further comprises: receiving a vehicle avoidance request sent by the avoidance object, wherein the vehicle avoidance request includes driving state information of the avoidance object; the driving state information includes at least one of position information, driving speed, driving direction, speed change information and vehicle lights; The driving route is dynamically adjusted based on the driving status information, so that the driver in the vehicle avoids the avoidance object based on the adjusted route.

7. The method according to any one of claims 1 to 6, characterized in that: The receiving of vehicle external audio data and vehicle external image data collected during vehicle driving includes: Collecting initial audio data of the vehicle during the driving process of the vehicle by means of a microphone disposed outside the vehicle; Performing sound signal preprocessing on the initial audio data of the vehicle to obtain the external audio data of the vehicle; the sound signal preprocessing includes equalization and / or denoising processing; The vehicle external image data is acquired through the panoramic imaging system of the vehicle; the vehicle external image data includes an image of at least one orientation.

8. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images