Obstacle prompting method and vehicle-mounted audio system

Abstract

The invention provides an obstacle prompting method and a vehicle-mounted audio system, and the method comprises the steps: determining the prompt content of an obstacle and a model tag based on the data of the obstacle collected by a vehicle sensor; constructing a sound source model of the obstacle based on a model corresponding to the model label; and generating a prompt tone of the obstacle based on the sound source model of the obstacle and the prompt content. The sound source of the obstacle outside the vehicle is modeled, so that prompt tone rendering for indicating parameters such as the direction, the size and the distance of the obstacle is realized, the obstacle prompt effect is improved, and the driving safety is improved; and meanwhile, the prompt tone of the obstacle is adaptively determined by the obstacle, so that the interestingness of obstacle prompt is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to an obstacle prompting method and a vehicle-mounted audio system. Background Art

[0002] With the continuous advancement of vehicle-mounted sensor technology, the types of vehicle-mounted sensors are becoming more and more abundant, and the accuracy of sensors is becoming higher and higher, which can sense obstacles around the vehicle and provide obstacle warnings.

[0003] In vehicle warning technology, the warning sound is usually monophonic audio data, and the warning content is fixed, such as a simple beeping sound. The warning method is monotonous and cannot play a sufficient warning role, resulting in poor driving safety.

[0004] Therefore, there is an urgent need to provide a rich obstacle warning solution with good warning effect to improve driving safety. Summary of the invention

[0005] The present application provides an obstacle warning method and an in-vehicle audio system, which improves the warning effect and driving safety by modeling the sound source of the obstacle and rendering a warning sound that shows the position of the obstacle. At the same time, the warning content that matches the obstacle is used for warning, which improves the richness and fun of the warning.

[0006] In a first aspect, the present application provides an obstacle warning method, comprising: determining the warning content and model label of the obstacle based on the obstacle data collected by the vehicle sensor; constructing a sound source model of the obstacle based on the model corresponding to the model label; and generating a warning sound of the obstacle based on the sound source model of the obstacle and the warning content.

[0007] In some embodiments, based on data collected by vehicle sensors, determining the prompt content and model label of the obstacle includes: determining attribute information of the obstacle based on the data collected by the vehicle sensors, the attribute information being used to describe at least one of the type, size, material, temperature, moving speed, position relative to the vehicle, distance from the vehicle, and sound absorption coefficient of the obstacle; acquiring audio data collected by a microphone; and determining the prompt content and model label of the obstacle based on the audio data and the attribute information.

[0008] In some embodiments, obtaining audio data collected by a microphone, and determining the prompt content and model label of the obstacle based on the audio data and the attribute information includes:

[0009] If the obstacle is a silent object, the prompt content and the model label of the obstacle are determined based on the attribute information; if the obstacle is a sound-emitting object, the audio data collected by the microphone is obtained to determine whether the audio data corresponding to the obstacle is included in the audio data; if so, the prompt content of the obstacle is determined based on the audio data corresponding to the obstacle, and the model label of the obstacle is determined based on the attribute information; if not, the prompt content and the model label of the obstacle are determined based on the attribute information.

[0010] In some embodiments, based on the attribute information, the model label of the obstacle is determined, including: if the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is greater than a first distance, then determining that the model label corresponding to the obstacle is a far-field low-speed model label; if the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is less than or equal to the first distance, then determining that the model label corresponding to the obstacle is a near-field low-speed model label.

[0011] In some embodiments, determining the prompt content of the obstacle based on the attribute information includes: determining the prompt scene of the obstacle based on the attribute information; and determining the prompt content of the obstacle from a plurality of pre-made standard prompt contents based on the prompt scene.

[0012] In some embodiments, when the obstacle is a silent object, the model label of the obstacle and the prompt content of the obstacle are determined based on the attribute information, including: if the obstacle is determined to be a fast-approaching obstacle based on the distance between the obstacle and the vehicle and the speed of the obstacle, and the distance between the obstacle and the vehicle is less than the braking safety distance corresponding to the current speed of the vehicle, then the model label corresponding to the obstacle is determined to be a high-speed collision label, and the prompt content corresponding to the obstacle is a high-speed collision prompt content; if the obstacle is a traffic light, then the model label corresponding to the obstacle is determined to be a traffic light label, and the prompt content corresponding to the obstacle is a traffic light prompt content; if the vehicle is in a reversing state, the obstacle is located behind the vehicle, and the distance between the obstacle and the vehicle is less than a second distance, then the model label corresponding to the obstacle is determined to be a reversing label, and the prompt content corresponding to the obstacle is a reversing prompt content.

[0013] In some embodiments, based on the model corresponding to the model label, a sound source model of the obstacle is constructed, including: based on data collected by vehicle sensors, determining the coordinates of the obstacle in the coordinate system of the vehicle; performing coordinate transformation on the obstacle, converting the coordinates of the obstacle into coordinates in a spherical coordinate system with the listening position as the origin; based on the converted coordinates of the obstacle, using the model corresponding to the model label, to construct a sound source model of the obstacle.

[0014] In some embodiments, based on the model corresponding to the model label, a sound source model of the obstacle is constructed, including: based on the geometric contour of the obstacle and based on multiple point sound sources, fitting the sound source corresponding to the obstacle to obtain a fitted sound source; based on the model corresponding to the model label, modeling the fitted sound source to obtain the sound source model of the obstacle.

[0015] In some embodiments, based on the model corresponding to the model label, a sound source model of the obstacle is constructed, including: performing spatial mapping projection on the obstacle to obtain a projection of the obstacle on the outer surface of the vehicle to obtain an obstacle projection; and modeling the obstacle projection based on the model corresponding to the model label to obtain a sound source model of the obstacle.

[0016] In some embodiments, the method also includes: determining a rendering algorithm label based on the model label, the type of obstacle, or the type of prompt content; rendering the prompt sound based on the rendering algorithm corresponding to the rendering algorithm label, and obtaining the gain of each speaker channel of the vehicle when rendering the prompt sound, so as to play the rendered prompt sound through each speaker of the vehicle.

[0017] In some embodiments, a rendering algorithm label is determined based on the model label, the type of the obstacle, or the type of the prompt content, including: if the model label is a traffic light label or the obstacle is a traffic light, then the rendering algorithm label is determined to be a traffic light rendering label, and the rendering algorithm corresponding to the traffic light rendering label is a sound bed rendering algorithm or a panoramic sound rendering algorithm; if the obstacle is a moving object, then the rendering algorithm label is determined to be a moving object rendering label, and the rendering algorithm corresponding to the moving object rendering label is a translation algorithm or a wave field synthesis rendering algorithm.

[0018] In a second aspect, the present application also provides a vehicle-mounted audio system, which is arranged in a vehicle, and includes: an obstacle warning sound generating device, a sensor and a renderer; the sensor is used to collect data of obstacles outside the vehicle; the obstacle warning sound generating device is used to execute the obstacle warning method provided in any embodiment of the present application to generate the warning sound of the obstacle; the renderer is used to render the warning sound so as to play the warning sound through the speaker array of the vehicle.

[0019] In a third aspect, the present application also provides an obstacle warning device, including: a content and label determination module, used to determine the warning content and model label of the obstacle based on the obstacle data collected by the vehicle sensor; a sound source modeling module, used to construct a sound source model of the obstacle based on the model corresponding to the model label; based on the sound source model of the obstacle and the warning content, generate a warning sound of the obstacle.

[0020] In a fourth aspect, the present application also provides an electronic device, comprising a memory and at least one processor, wherein the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the method provided in the first aspect.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method provided in the first aspect is implemented.

[0022] In a sixth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which implements the method provided in the first aspect when executed by a processor.

[0023] The obstacle prompt method and vehicle audio system provided by the present application, based on the obstacle data collected by the vehicle sensor, personalize the prompt content for the obstacle, thereby improving the richness of the obstacle prompt; at the same time, based on the obstacle data, determine the model label of the sound source model used in obstacle sound source modeling, thereby realizing the sound source modeling of the obstacle and obtaining the sound source model of the obstacle; when rendering the obstacle prompt content, the obstacle sound source model is combined for rendering, and a prompt sound that can display parameters such as the obstacle's orientation, size, and distance from the vehicle is rendered, thereby greatly improving the effect of the obstacle prompt, reducing the possibility of collision between the vehicle and the obstacle, and improving the safety of vehicle driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the architecture of a 7.1.4-channel in-vehicle audio system provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of an obstacle prompting method provided in an embodiment of the present application;

[0028] Figure 4 For this application Figure 3 A schematic diagram of the rendering effect of the obstacle warning sound in the implemented embodiment;

[0029] Figure 5 A flowchart of another obstacle prompting method provided in an embodiment of the present application;

[0030] Figure 6 A mapping relationship diagram between the prompt content provided in the embodiment of the present application and the model used for modeling;

[0031] Figure 7 A schematic diagram of obstacle space mapping projection provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a vehicle audio system provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of an obstacle warning system provided in an embodiment of the present application.

[0034] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, the vehicle is provided with an ADAS (Advanced Driving Assistance System). After the vehicle is started, the ADAS on the vehicle uses sensors installed on the vehicle, such as cameras, radars, etc., to sense the environment around the vehicle and collect environmental data to achieve obstacle detection.

[0037] ADAS detects obstacles such as Figure 1 The direction of the pedestrians, vehicles and pedestrians is as follows: Figure 1As shown, a warning is issued through the speakers on the vehicle, such as controlling the center speaker to emit a short beeping sound.

[0038] The above obstacle warning method is single, and the driver can only know that there are obstacles around the vehicle through the prompt sound (such as beep) emitted by the speaker. The driver needs to further determine the location of the obstacle, the distance from the vehicle and other information through the rear mirror, reversing image, etc., which is not conducive to driving safety. When the visibility of the vehicle is low, the driver will not be able to confirm the location of the obstacle, and it is easy to collide with the obstacle, and the driving safety is poor.

[0039] With the continuous development of in-car audio systems, the channel layout of in-car audio systems has developed from traditional stereo to three-dimensional sound, such as 7.1.4 channels, 7.3.4 channels, etc.

[0040] Figure 2 The schematic diagram of the architecture of the 7.1.4-channel in-vehicle audio system provided in the embodiment of the present application is as follows: Figure 2 As shown in the figure, the 7.1.4-channel car audio system includes 7 horizontal speakers (or surround speakers), 1 subwoofer and 4 sky speakers (or top speakers). Figure 1 Mid-level speakers are represented by solid dots, subwoofers are represented by dashed circles, and overhead speakers are represented by solid circles.

[0041] When rendering the audio signal, the three-dimensional sound rendering technology is used to control the gain of each speaker channel in the speaker array of the car audio system to achieve immersive spatial sound effects.

[0042] Based on this, the present application provides an obstacle prompt method, which realizes obstacle sound source modeling and differentiated matching of obstacle prompt content through data collected by vehicle sensors, and renders the obstacle prompt content in combination with the obstacle sound source model to render a prompt sound that prompts the obstacle attribute information. The attribute information includes but is not limited to direction, size, moving speed, distance from the vehicle, etc., thereby improving the effect of obstacle prompts and driving safety, and through differentiated obstacle prompts, enriching the methods of obstacle prompts and improving the fun of obstacle prompts.

[0043] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0044] Figure 3This is a flow chart of an obstacle prompting method provided in an embodiment of the present application. The method is applied to a vehicle and can be executed by an onboard audio system or obstacle prompting system on the vehicle, or by a server or other remote device. Figure 3 As shown, the obstacle prompting method comprises the following steps:

[0045] Step S301, based on the obstacle data collected by the vehicle sensor, determine the obstacle prompt content and model label.

[0046] Sensors on the vehicle include but are not limited to infrared sensors, image sensors, millimeter wave radar sensors, lidar sensors, ultrasonic sensors, etc.

[0047] The vehicle is provided with a speaker array, and the number of speakers in the speaker array is at least 3.

[0048] The model label is the label of the model used to establish the sound source model of the obstacle.

[0049] The prompt content can be in the form of text or audio.

[0050] During the driving process of the vehicle, the sensors installed on the vehicle collect data about the surrounding environment. When it is sensed that there is an obstacle in the vehicle's surrounding environment, the obstacle data collected by the sensor is obtained, and based on the obstacle data, prompt content and model labels are matched for the obstacle.

[0051] Obstacles can be stationary objects, such as traffic lights, guardrails, trees, walls, water barriers, etc., or moving objects, such as pedestrians, other vehicles, etc. Obstacles can be sound-emitting objects or silent objects.

[0052] Specifically, the attribute information of the obstacle may be determined based on the obstacle data collected by the sensor, and the prompt content and the model label of the obstacle may be determined based on the attribute information of the obstacle.

[0053] The attribute information includes but is not limited to the type, size, material, temperature, moving speed, position relative to the vehicle, distance from the vehicle, and sound absorption coefficient of the obstacle.

[0054] The obstacle's size, distance from the obstacle to the vehicle and other attribute information can be determined based on obstacle data collected by a millimeter-wave radar sensor or a lidar sensor installed on the vehicle.

[0055] At night or in other low-visibility environments, the size of the obstacle, the distance between the obstacle and the vehicle, the temperature of the obstacle and other attribute information can be determined based on the data collected by the infrared sensor installed on the vehicle.

[0056] Based on the image of the obstacle captured by the image sensor installed on the vehicle, such as a monocular / binocular camera, the obstacle type, obstacle size, distance between the obstacle and the vehicle, obstacle material and other attribute information can be identified.

[0057] Based on the obstacle data collected by the ultrasonic sensor installed on the vehicle, attribute information such as the distance between the obstacle and the vehicle and the sound absorption coefficient of the obstacle can be determined.

[0058] In some embodiments, the sensor may further include a microphone, and audio data of the obstacle may be collected based on the microphone on the vehicle to generate prompt content of the obstacle based on the audio data of the obstacle.

[0059] In some embodiments, the category of the obstacle, such as pedestrians, dogs, multimedia devices, etc., can also be determined based on the audio data of the obstacle.

[0060] In some embodiments, the prompt content of the obstacle can be generated based on the attribute information of the obstacle and a pre-trained generation network model, and the model label of the obstacle can be determined based on the attribute information of the obstacle and a pre-trained classification network model.

[0061] Exemplarily, the generation network model may be a recurrent neural network model (RNN), a generative adversarial network model (GAN), or other network models. The classification network model may be a convolutional neural network (CNN), a recurrent neural network model (RNN), a Transformer attention model, or other network models.

[0062] In other embodiments, the obstacle scene may be determined based on the attribute information of the obstacle, and the prompt content and model label of the obstacle may be determined based on the obstacle scene.

[0063] Based on the attribute information of various possible obstacles, multiple obstacle scenes can be divided in advance, the scene conditions that the attribute information should meet in each obstacle scene can be summarized, and matching prompt content and model labels can be set for each obstacle scene. After obtaining the attribute information of the obstacles around the vehicle, the scene conditions that the attribute information of the obstacle meets are determined, and the obstacle scene corresponding to the obstacle is obtained, and the prompt content and model label matched by the corresponding obstacle scene are used as the prompt content and model label of the obstacle.

[0064] Optionally, based on the data collected by the vehicle sensor, the prompt content and the model label of the obstacle are determined, including: based on the data collected by the vehicle sensor, determining the attribute information of the obstacle; obtaining audio data collected by the microphone; based on the audio data and the attribute information, determining the prompt content and the model label of the obstacle.

[0065] Specifically, it is possible to determine whether the audio data collected by the microphone includes the sound emitted by the obstacle. If so, the audio data corresponding to the obstacle is extracted from the audio data, and the prompt content is generated based on the audio data corresponding to the obstacle. If the audio data collected by the microphone does not include the sound emitted by the obstacle, the prompt content of the obstacle is determined based on the attribute information of the obstacle.

[0066] The model label of the obstacle may be determined based only on the attribute information of the obstacle, or may be determined in combination with the attribute information of the obstacle and audio data corresponding to the obstacle collected by a microphone.

[0067] When the obstacle is a sound-emitting obstacle, the obstacle prompt content can be directly generated based on the audio data of the obstacle collected by the microphone. The audio data of the obstacle collected by the microphone can be subjected to noise reduction processing to obtain the obstacle prompt content. When the obstacle is a silent obstacle, the obstacle prompt content can be determined based on the attribute information of the obstacle.

[0068] For example, when the audio data corresponding to the obstacle collected by the microphone is the sound of a pedestrian shouting, the prompt content of the pedestrian (obstacle) is determined to be the content of the shouting. Based on the shouting sound collected by the microphone, the category of the obstacle is determined to be a pedestrian, and the model label of the pedestrian is determined in combination with the distance between the pedestrian and the vehicle, the orientation relative to the vehicle, etc., such as the near-field pedestrian label, where the near-field indicates that the distance between the pedestrian and the vehicle is relatively close, and the orientation of the pedestrian relative to the vehicle is used to indicate the coordinates of the sound source in the sound source model corresponding to the pedestrian.

[0069] In some embodiments, data collected by sensors installed on the vehicle can be obtained at a certain period, and based on the data collected during the period, it can be determined whether there is an obstacle; if so, the prompt content and model label of the obstacle can be determined based on the data collected during the period. The period can be a fixed period, such as 1s, or a period determined based on the vehicle speed. The faster the vehicle speed, the shorter the period.

[0070] Step S302: construct a sound source model of the obstacle based on the model corresponding to the model label.

[0071] After the model label of the obstacle is determined, a sound source model of the obstacle is constructed based on the model or modeling algorithm corresponding to the model label.

[0072] Exemplarily, the model corresponding to the model label may be a combination of one or more of a plane sound source model, a point sound source model, a spherical sound source model, a columnar sound source model, a linear sound source model, a far-field sound source model, a near-field sound source model, a high-speed sound source model, and a low-speed sound source model. Taking the combination of a point sound source model and a far-field sound source model as an example, the combined model may be called a far-field point sound source model, and taking the combination of a columnar sound source model, a near-field sound source model, and a high-speed sound source model as an example, the combined model may be called a high-speed near-field columnar sound source model.

[0073] The sound source model of the obstacle is a stereo sound source model. Based on the three-dimensional geometry of the obstacle and utilizing the propagation characteristics of the sound, the stereo sound source model of the obstacle can be established by using a model corresponding to the model label, such as a far-field point sound source model.

[0074] The three-dimensional geometry of the obstacle can be obtained through data collected by sensors, such as by identifying obstacle images, through data collected by radar sensors, etc.

[0075] The sound source coordinates of the stereo sound source corresponding to the obstacle can be determined based on the obstacle's relative position to the vehicle, the distance between the obstacle and the vehicle, and other attribute information. A virtual sound source model, i.e. the sound source model of the obstacle, is established at the sound source coordinates through the model corresponding to the obstacle's model label. By rendering the virtual sound source model, a prompt sound indicating the obstacle's position is rendered.

[0076] In order to further improve the accuracy of the sound source model and the effect of obstacle warning, a stereo sound source model of the obstacle can be established by using the model corresponding to the model label based on the three-dimensional geometry of the obstacle, as well as the obstacle's moving speed, position relative to the vehicle, sound absorption coefficient and other attribute information, so that the rendered obstacle warning sound can prompt information such as the obstacle's moving speed and the obstacle's position relative to the vehicle.

[0077] Step S303: generating a prompt sound of the obstacle based on the sound source model of the obstacle and the prompt content.

[0078] When the prompt content is text, the prompt content is converted into an audio format.

[0079] The obstacle sound source model is used as the sound source, and the obstacle prompt content is used as the sound emitted by the sound source to obtain the obstacle prompt sound. The prompt sound is rendered and played through the speaker array in the vehicle to simulate the sound of the prompt content emitted by the sound source reaching the listening position, so as to realize the prompting of the obstacle position through the obstacle prompt sound. The listening position can be the position of the driver or passenger in the vehicle, or a designated position in the vehicle.

[0080] During rendering, the gain of each speaker channel can be determined based on the position of each speaker in the speaker array in the vehicle, so that the sound synthesized by the prompt sound emitted by each speaker at the listening position is close to the prompt content emitted by the sound source model of the obstacle heard at the listening position. The listening position can be the position of the driver or passenger's head.

[0081] Since the obstacle's position relative to the vehicle, distance from the vehicle and other attribute information are taken into consideration when modeling the obstacle sound source, the rendered prompt sound is the sound emitted by a virtual sound source at the corresponding position of the obstacle, thereby achieving the effect of indicating the obstacle's position based on the prompt sound.

[0082] For example, taking the obstacle as a wall, a rendered prompt sound such as "Please note there is a wall" can be used to enable listeners, such as drivers and passengers, to quickly determine the approximate location of the wall based on the prompt sound, such as the left front of the vehicle.

[0083] Figure 4 For this application Figure 3 A schematic diagram of the rendering effect of the obstacle warning sound in the implemented embodiment, as shown in FIG. Figure 4 As shown, there is an obstacle in front of the right side of the vehicle ( Figure 4 The obstacle is represented by a solid circle in the figure), such as a pedestrian crossing the road. Based on the distance between the obstacle and the vehicle, the direction relative to the vehicle, and the size of the obstacle, the prompt content is determined to be "Beware of obstacles", and a sound source model of the obstacle is established. By rendering the speaker of the vehicle, the sound of "Beware of obstacles" emitted by the speaker reaching the listening position simulates the sound source model of the pedestrian. This sound source emits the sound of "Beware of obstacles" reaching the listening position, so that the user at the listening position, such as the driver, can determine the location of the obstacle after hearing the prompt sound of "Beware of obstacles". Figure 4 The dotted circle in the figure indicates the position of the obstacle determined by the user.

[0084] When the channel layout of the speakers in the vehicle is a three-dimensional channel layout, based on the three-dimensional channel rendering technology, the sound source model of the obstacle can be used as a virtual sound source to render a three-dimensional prompt sound corresponding to the prompt content of the obstacle, thereby further improving the accuracy of the obstacle position indicated by the prompt sound.

[0085] In the technical solution provided in this embodiment, based on the obstacle data collected by the vehicle sensor, the prompt content is personalized for the obstacle to improve the richness of the obstacle prompt; at the same time, the model label of the sound source model used in obstacle sound source modeling is determined based on the obstacle data, thereby realizing the sound source modeling of the obstacle and obtaining the sound source model of the obstacle; when rendering the obstacle prompt content, the obstacle sound source model is combined for rendering to render a prompt sound that can display parameters such as the obstacle's orientation, size, and distance from the vehicle, thereby greatly improving the effect of the obstacle prompt, reducing the possibility of collision between the vehicle and the obstacle, and improving the safety of vehicle driving.

[0086] Figure 5 A flowchart of another obstacle prompting method provided in an embodiment of the present application. Figure 3 Based on the embodiment shown, step S301 and step S303 are further refined, and relevant steps determined by the rendering algorithm are added before step S303, such as Figure 5 As shown, the obstacle prompting method provided in this embodiment specifically includes the following steps:

[0087] Step S501: determining the attribute information of the obstacle based on the data collected by the vehicle sensor.

[0088] Specifically, the type and size of the obstacle can be determined based on the obstacle image collected by the image sensor. When the image sensor is a 3D camera or a binocular camera, the three-dimensional coordinates of the obstacle in the vehicle coordinate system can also be determined based on the collected obstacle image. The vehicle coordinate system can be a coordinate system with the left front wheel of the vehicle as the origin and the driving direction of the vehicle as the positive direction of the longitudinal axis. Based on the data collected by one or more of the infrared sensor, radar sensor, ultrasonic sensor, etc., the direction and distance of the obstacle relative to the vehicle are determined.

[0089] After obtaining obstacle-related data collected by vehicle sensors, such as obstacle images, relevant recognition algorithms, such as image recognition algorithms, can be used to identify the type, material, size, and position relative to the vehicle of the obstacle. By processing the data collected by the sensor, such as clustering, the size or size of the obstacle, the position of the obstacle relative to the vehicle, and the moving speed or approaching speed of the obstacle can be determined. Based on the ultrasonic waves returned by the obstacle collected by the ultrasonic sensor, the distance between the obstacle and the vehicle can be determined, and the sound absorption coefficient of the obstacle can also be determined.

[0090] Step S502: If the obstacle is a silent object, a prompt content of the obstacle is determined based on the attribute information, and a model label of the obstacle is determined based on the attribute information.

[0091] The silent object may include a wall, a guardrail, a traffic light, etc. Based on the category of the obstacle, it may be determined whether the obstacle is a sound-emitting object or a silent object.

[0092] When the obstacle is a silent object, such as a branch, the prompt content and model label of the obstacle are determined based on the attribute information of the obstacle.

[0093] The deep learning model can be used to determine the obstacle prompt content and model label based on the attribute information of the obstacle.

[0094] Optionally, determining the prompt content of the obstacle based on the attribute information includes: determining a prompt scene of the obstacle based on the attribute information; and determining the prompt content of the obstacle from a plurality of pre-made standard prompt contents based on the prompt scene.

[0095] One or more thresholds may be set for the attribute information, and based on the comparison result between the attribute information and the corresponding threshold, the prompt scene of the obstacle is determined, thereby selecting one from pre-made standard prompt contents as the prompt content of the obstacle based on the prompt scene.

[0096] For obstacle categories, various categories may be grouped in advance, and different groups correspond to different prompting scenes. Thus, after the obstacle category is determined, the obstacle prompting scene is determined based on the group to which the obstacle category belongs.

[0097] The prompting scene may include a traffic light prompting scene, a pedestrian passing prompting scene, a shouting prompting scene, a high-speed collision prompting scene, etc. The type of obstacle in the traffic light prompting scene is a traffic light. In the pedestrian passing prompting scene, the type of obstacle is a pedestrian, and the obstacle moves at a low speed. The shouting prompting scene is a scene corresponding to an obstacle that is a sound-emitting object. In the high-speed collision prompting scene, the obstacle moves at a high speed.

[0098] The prompt scenario may also include other scenarios, which are not limited in this application.

[0099] Optionally, when the obstacle is a silent object, the model label of the obstacle and the prompt content of the obstacle are determined based on the attribute information, including: if the obstacle is determined to be a rapidly approaching obstacle based on the distance between the obstacle and the vehicle and the speed of the obstacle, and the distance between the obstacle and the vehicle is less than the braking safety distance corresponding to the current speed of the vehicle, then the model label corresponding to the obstacle is determined to be a high-speed collision label, and the prompt content corresponding to the obstacle is a high-speed collision prompt content; if the obstacle is a traffic light, then the model label corresponding to the obstacle is determined to be a traffic light label, and the prompt content corresponding to the obstacle is a traffic light prompt content; if the vehicle is in a reversing state, the obstacle is located behind the vehicle, and the distance between the obstacle and the vehicle is less than a second distance, then the model label corresponding to the obstacle is determined to be a reversing label, and the prompt content corresponding to the obstacle is a reversing prompt content.

[0100] Specifically, when the moving speed of the obstacle relative to the vehicle is greater than a first speed, such as 30 km / h, 50 km / h or other values, and the distance between the obstacle and the vehicle is less than a preset distance, such as 8 m, 10 m or other values, the obstacle is determined to be a fast approaching obstacle.

[0101] When the obstacle is a fast approaching obstacle, it is determined that the model label corresponding to the obstacle is a fast moving label, and the corresponding prompt content is a fast moving prompt content.

[0102] Exemplarily, the model corresponding to the model label of the fast moving label may be a far-field sound source model.

[0103] When the obstacle is approaching quickly, the distance between the obstacle and the vehicle is continuously monitored. When the distance between the obstacle and the vehicle is less than the braking safety distance corresponding to the current speed of the vehicle, the model label corresponding to the obstacle is determined to be a high-speed collision label, and the prompt content corresponding to the obstacle is a high-speed collision prompt content.

[0104] Exemplarily, the model corresponding to the high-speed collision label model label may be a far-field sound source model with Doppler effect added.

[0105] The prompt content corresponding to the traffic light, that is, the traffic light prompt content, can be generated by identifying the traffic light signal, such as "the red light will turn green soon", "the red light has 5 seconds left", "the traffic light is green, you can pass", etc.

[0106] Exemplarily, the model corresponding to the model label of the traffic light label may be a columnar sound source model.

[0107] When the vehicle is in reverse, the detected obstacles are usually close to the vehicle and the vehicle speed is also low. If an obstacle is detected behind the vehicle, including behind the rear mirror of the vehicle, such as the rear, left rear, and right rear, when the distance between the obstacle and the vehicle is less than the second distance, such as 1m, 0.5m or other smaller distances, the model label of the obstacle is a reverse label, and the prompt content is a reverse prompt content.

[0108] For example, the model corresponding to the model label of the reversing label may be a near-field sound source model. The reversing prompt content may be "Beware of obstacles behind the vehicle" or "Beware of O behind the vehicle", and "O" may be determined based on the type of obstacle, such as a cat, a pedestrian, another vehicle, a pillar, etc.

[0109] Step S503: if the obstacle is a sound-emitting object, then the audio data collected by the microphone is obtained.

[0110] When it is determined that there is an obstacle based on the data collected by the sensor, it is first determined whether the obstacle is a sound-emitting object. If so, the audio data collected by the microphone a period of time before the current time is obtained, such as the previous 2s, 3s or other time periods.

[0111] Step S504: determine whether the audio data includes audio data corresponding to the obstacle.

[0112] Based on the audio data collected by the microphone, it can be determined whether the audio data includes audio data corresponding to the obstacle.

[0113] Specifically, based on the attribute information of the obstacle, such as one or more of its category, distance from the vehicle, position relative to the vehicle, moving speed, etc., it can be determined whether the audio data collected by the microphone includes audio data corresponding to the obstacle.

[0114] If the duration of the audio data corresponding to the obstacle collected by the microphone is too short, such as less than 0.5s, 0.2s, 0.1s, etc., the audio data corresponding to the obstacle is not included in the audio data collected by the microphone.

[0115] When the obstacle is a pedestrian, voiceprint recognition technology can be used to determine whether there is audio data of pedestrians in the audio data collected by the microphone. In order to avoid mistaking the voice of passengers or drivers in the vehicle for the voice of pedestrians, the intensity of the collected audio data can be combined for judgment, or the audio data of passengers or drivers in the vehicle can be filtered out by comparing it with the voiceprint collected by the microphone in the past.

[0116] Step S505: If yes, then based on the audio data corresponding to the obstacle, determine the prompt content of the obstacle, and based on the attribute information, determine the model label of the obstacle.

[0117] Specifically, if the audio data includes audio data corresponding to the obstacle, the audio data corresponding to the obstacle is extracted from the audio data, and the audio data corresponding to the obstacle is processed into prompt content of the obstacle, and the model label of the obstacle is determined based on the attribute information of the obstacle. The processing of the obstacle includes but is not limited to noise reduction, text recognition, format conversion, etc.

[0118] The model label of the obstacle can be determined based on the value range of each attribute information of the obstacle.

[0119] In some embodiments, label conditions that the attribute information under each model label should satisfy can be pre-established, so that based on the values ​​of each attribute information of the obstacle, the model label whose attribute information of the obstacle satisfies the label conditions is regarded as the model label of the obstacle.

[0120] Based on one or more pieces of attribute information of the obstacle, the model labels corresponding to the one or more pieces of attribute information may be determined, and based on the model labels corresponding to each piece of attribute information of the obstacle, the model label of the obstacle may be determined.

[0121] Exemplarily, when the obstacle moves too fast, the model tag corresponding to the obstacle's moving speed can be determined as a high-speed moving tag. When the obstacle moves slowly, the model tag corresponding to the obstacle's moving speed can be determined as a low-speed moving tag. When the obstacle is far away from the vehicle, the model tag corresponding to the attribute information of the distance between the obstacle and the vehicle can be determined as a far-field tag; when the obstacle is close to the vehicle, the model tag corresponding to the attribute information of the distance between the obstacle and the vehicle can be determined as a near-field tag.

[0122] The attribute information also includes the outline of the obstacle, and based on the outline of the obstacle, a model label corresponding to the outline of the obstacle can be determined.

[0123] Exemplarily, when the outline of the obstacle is a columnar outline, the model label of the obstacle is a columnar model label, so that the sound source model of the obstacle is generated based on the columnar sound source model. If the outline of the obstacle is linear, such as a guardrail, the model label of the obstacle is a linear model label, so that the sound source model of the obstacle is generated based on the linear sound source model.

[0124] Taking the obstacle as a fast-moving vehicle flow as an example, the model label of the obstacle may be a fast-moving linear model label (a combination of a fast-moving label and a linear model label).

[0125] Optionally, determining a model label of the obstacle based on the attribute information includes:

[0126] If the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is greater than the first distance, the model label corresponding to the obstacle is determined to be a far-field low-speed model label; if the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is less than or equal to the first distance, the model label corresponding to the obstacle is determined to be a near-field low-speed model label.

[0127] Pedestrians usually move at a low speed, so when the obstacle is a pedestrian, it can be directly determined as a low-speed scene. Combined with the comparison result between the distance between the pedestrian and the vehicle and the first distance, the model label of the pedestrian is determined to be a far-field low-speed model label or a near-field low-speed model label.

[0128] Exemplarily, the first distance may be 1.5 m, 2 m, 2.5 m or other distances.

[0129] Step S506: if not, determining the prompt content of the obstacle based on the attribute information, and determining the model label of the obstacle based on the attribute information.

[0130] In some embodiments, in addition to determining whether the audio data includes audio data corresponding to the obstacle, it can also be determined whether the total duration of the audio data corresponding to the obstacle is greater than or equal to a preset duration. If so, the prompt content of the obstacle is determined based on the audio data corresponding to the obstacle, and the model label of the obstacle is determined based on the attribute information of the obstacle; if not, that is, the audio data corresponding to the obstacle is not included, or the total duration of the audio data corresponding to the obstacle is less than the preset duration, the prompt content and model label of the obstacle are determined based on the attribute information of the obstacle.

[0131] If the obstacle is a silent object, or the microphone does not collect audio data of the obstacle, or the duration of the audio data corresponding to the obstacle collected by the microphone is too short, the prompt content and model label of the obstacle are determined based on the attribute information of the obstacle.

[0132] Step S507: construct a sound source model of the obstacle based on the model corresponding to the model label.

[0133] After determining the model label of the obstacle, a stereo sound source model of the obstacle can be constructed based on the model corresponding to the model label and according to data such as the coordinates, outline, size, material, sound absorption coefficient, and distance from the vehicle of the obstacle.

[0134] Taking a wall as an example, the model label of the wall is a plane sound source label, and the model corresponding to the plane sound source label is a plane sound source model. Based on the distance between the wall and the vehicle and the size of the wall, a plane sound source model corresponding to the wall is constructed.

[0135] The planar sound source model can use Green's Function and Rayleigh integral to describe the dynamic propagation directivity of the planar sound source.

[0136] For an obstacle with a complex contour, the corresponding model label may be a point sound source fitting label, that is, the sound source model of the obstacle is obtained by fitting multiple point sound sources.

[0137] For obstacles that are far away from the vehicle, such as buildings and other vehicles, the corresponding model label is the far-field model label. The sound source model of the obstacle can be constructed through the far-field sound source model. The sound source model of the obstacle is modeled according to the sound generation and propagation characteristics of the far-field sound source.

[0138] For obstacles that are closer to the vehicle, such as pedestrians, the corresponding model label is the near-field model label. The sound source model of the obstacle can be constructed through the near-field sound source model. The sound source modeling of the obstacle is performed according to the sound generation and propagation characteristics of the near-field sound source.

[0139] Moving obstacles can be divided into high-speed moving obstacles and low-speed moving obstacles. The moving speed of the obstacle is relative to the moving speed of the vehicle.

[0140] When an obstacle approaches the vehicle at a relatively high speed (including the situation where the vehicle is traveling towards the obstacle at high speed), the model label of the obstacle is a high-speed moving label. The far-field sound source model with Doppler effect can be used to model the obstacle sound source. The introduction of the Doppler effect reflects the change in the speed of the sound source of the obstacle or the warning sound, thereby improving the realistic hearing of the warning sound and enhancing the driver's sense of crisis.

[0141] When an obstacle approaches the vehicle at a lower speed, the model label of the obstacle is a low-speed moving label. The near-field sound source model based on the obstacle contour can be used to model the sound source of the obstacle, combined with the propagation directionality of the sound emitted by the obstacle to highlight the characteristics of the obstacle, so that the driver in the car can better identify the object corresponding to the prompt sound.

[0142] For the traffic light obstacle, the model used models the traffic light prompt content as background sound to distinguish it from other obstacles. The sound source of the traffic light can also be modeled in combination with the characteristics of vortex sounds such as flowing wind and water sounds to enrich the dynamics and range of the traffic light sound source model.

[0143] In some embodiments, to facilitate sound source modeling, before performing sound source modeling on an obstacle, coordinate transformation may be performed on the obstacle to transform the coordinates of the obstacle into polar coordinates, specifically polar coordinates with the listening position as the origin.

[0144] Optionally, based on the model corresponding to the model label, a sound source model of the obstacle is constructed, including: based on data collected by vehicle sensors, determining the coordinates of the obstacle in the coordinate system of the vehicle; performing coordinate transformation on the obstacle, converting the coordinates of the obstacle into coordinates in a spherical coordinate system with the listening position as the origin; and based on the converted coordinates of the obstacle, using the model corresponding to the model label to construct a sound source model of the obstacle.

[0145] The listening position may be the position of any user in the vehicle, such as the position of the user's head, or may be a position specified by the user, such as the main driving seat.

[0146] Step S508: generating a prompt sound of the obstacle based on the sound source model of the obstacle and the prompt content.

[0147] Step S509: determining a rendering algorithm label based on at least one of the model label, the type of the obstacle, and the type of the prompt content.

[0148] In order to improve the rendering effect of the obstacle warning sound, the rendering algorithm can be adapted for the obstacle.

[0149] In some embodiments, the rendering algorithm label corresponding to the obstacle may be determined based on the type of the obstacle. The type of the obstacle may be determined based on the attribute information of the obstacle, such as the type of obstacle, the moving speed, and the like.

[0150] Exemplarily, the types of obstacles may include moving types, such as pedestrians, moving vehicles, and stationary types, such as traffic lights, guardrails, etc. Moving types may be further divided into high-speed moving types, fast moving types, and slow moving types.

[0151] In other embodiments, the rendering algorithm label of the obstacle may be determined based on the type of prompt content or the model label of the obstacle.

[0152] The correspondence between the model label, the type of obstacle, at least one of the type of prompt content, and the rendering algorithm label can be suggested in advance to determine the rendering algorithm label based on the determined model label of the obstacle, the type of obstacle, at least one of the type of prompt content, and the correspondence.

[0153] In some embodiments, the rendering algorithm labels can be divided into three categories: moving object category, fixed object category, and other category. The type of the rendering algorithm label is consistent with the type of the prompt content and the type of the obstacle.

[0154] Exemplarily, the types of prompt content corresponding to the rendering algorithm labels of the mobile object class may include pedestrian passing prompt content, high-speed collision prompt content, etc., the corresponding model labels may be high-speed moving model labels, low-speed moving model labels, etc., and the corresponding obstacle types are mobile obstacle types. The types of prompt content corresponding to the rendering algorithm labels of the fixed object class may include traffic light prompt content, guardrail prompt content, wall prompt content, etc., the corresponding model labels may be traffic light model labels, wall model labels, guardrail model labels, etc., and the corresponding obstacle types are fixed obstacle types. The types of prompt content corresponding to the rendering algorithm labels of other classes may include pedestrian shouting prompt content, guide dog barking prompt content, etc., the corresponding model labels may be shouting model labels, and the corresponding obstacle types are sounding obstacle types.

[0155] Optionally, based on the model label, the type of the obstacle and at least one of the prompt content, a rendering algorithm label is determined, including: if the model label is a traffic light label or the obstacle is a traffic light, then the rendering algorithm label is determined to be a traffic light rendering label, and the rendering algorithm corresponding to the traffic light rendering label is a sound bed rendering algorithm or a panoramic sound rendering algorithm; if the obstacle is a moving object, then the rendering algorithm label is determined to be a moving object rendering label, and the rendering algorithm corresponding to the moving object rendering label is a translation algorithm or a wave field synthesis rendering algorithm.

[0156] The sound bed rendering algorithm is a rendering algorithm proposed for the sound bed.

[0157] The panoramic sound rendering algorithm can be First Order Ambisonics (FOA) or High Order Ambisonics (HOA). The High Order Ambisonics rendering algorithm uses high-order spherical harmonic functions, such as second-order and third-order, to improve the spatial resolution of the rendering algorithm, enrich the details of the sound and image, and improve the stereo effect of the prompt sound.

[0158] The panning algorithm is a rendering algorithm that uses panning, such as the basic panning algorithm, which performs sound rendering through simple panning, and improved panning algorithms, such as the Vector Based Amplitude Panning (VBAP) and the Distance Based Amplitude Panning (DBAP).

[0159] Wave Field Synthesis (WFS) is a technology that uses a speaker array to reconstruct the spatial sound field and reproduce the real sound field at a specified listening position.

[0160] By differentiating the rendering algorithm for different obstacles, the three-dimensional rendering effect of the obstacle warning sound is improved, and a three-dimensional sound field can be realized in the car.

[0161] Step S510, rendering the prompt sound based on the rendering algorithm corresponding to the rendering algorithm tag, obtaining the gain of each speaker channel of the vehicle when rendering the prompt sound, so as to play the rendered prompt sound through each speaker of the vehicle.

[0162] In other embodiments, a default rendering algorithm may be used for rendering, such as the HOA algorithm, the VBAP algorithm, and the like.

[0163] In this embodiment, the scene corresponding to the obstacle is subdivided. For the scene where a sound-emitting object is present and the vehicle microphone collects the audio data of the sound-emitting object, the obstacle prompt content is obtained based on the audio data of the sound-emitting object collected by the microphone, thereby improving the authenticity of the prompt content. For the scene where a sound-emitting object is not present or the microphone does not collect the audio data of the obstacle or the collected audio data of the obstacle is too short, the prompt content matching the obstacle attribute is automatically generated based on the attribute information of the obstacle, thereby enriching the style of the prompt content and improving the interest of the obstacle prompt. In order to improve the accuracy of obstacle sound source modeling, adaptive sound source modeling based on obstacle attribute information is implemented, and a sound source model of the obstacle is constructed by using a model matching the obstacle attribute. The prompt sound of the obstacle is rendered in the vehicle in a manner that simulates the sound source model to issue the prompt content, so that the driver or passenger can determine the position of the obstacle through the prompt sound, thereby achieving the effect of identifying the position by listening to the sound. When the visibility is low, the position of the obstacle is prompted by the obstacle prompt sound provided by the application, thereby greatly reducing the possibility of collision between the vehicle and the obstacle and improving driving safety. At the same time, through the adaptive selection of the rendering algorithm, the three-dimensional effect of the obstacle warning sound is further improved, and the warning effect is improved.

[0164] For example, Figure 6 A mapping relationship diagram between the prompt content provided in the embodiment of the present application and the model used for modeling, such as Figure 6As shown in the figure, the prompt content includes shouting prompt content (corresponding to the scene where the sound object and the microphone collects the audio data of the obstacle), pedestrian passing prompt content (the obstacle is a pedestrian passing by), traffic light prompt content (the obstacle is a traffic light), high-speed collision prompt content (the obstacle moves at a fast speed and is close to the vehicle), reversing prompt content (the vehicle is in a reversing state, the obstacle is located behind the vehicle and is close to the vehicle), etc. The model is divided into multiple types according to the distance of the sound source, solid geometry, relative speed (relative to the speed of the vehicle), whether it is audible, etc., and is divided into near-field model and far-field model according to the distance of the sound source. According to the solid geometry modeling, it is divided into spherical sound source model, plane sound source model, columnar sound source model, linear sound source model, etc. According to the relative speed modeling, it is divided into high-speed moving sound source model and low-speed moving sound source model, and also includes traffic light sound source model and sound object sound source model. When modeling the sound source model of the sound object, it is also necessary to perform voice recognition and directivity measurement of the sound object.

[0165] Continue to see Figure 6 In order to speed up the sound source modeling, the sound source modeling can also be accelerated, such as by directivity fitting and / or sound source projection.

[0166] Directivity fitting is to fit the sound source of obstacles through multiple point sound sources. Sound source projection is to project the obstacles onto the outer surface of the vehicle by spatial mapping projection, taking the listening position in the vehicle as the source point.

[0167] Optionally, based on the model corresponding to the model label, a sound source model of the obstacle is constructed, including: based on the geometric contour of the obstacle and based on multiple point sound sources, fitting the sound source corresponding to the obstacle to obtain a fitted sound source; based on the model corresponding to the model label, modeling the fitted sound source to obtain the sound source model of the obstacle.

[0168] The geometric outline of the obstacle can be determined by obstacle data collected by sensors such as radar sensors, image sensors, etc. Directivity fitting of the curve corresponding to the geometric outline of the obstacle is performed through multiple point sound sources, which realizes the use of multiple point sound sources to replace obstacles for sound source modeling and reduces the amount of calculation.

[0169] In order to reduce the amount of calculation for sound source modeling and prompt sound rendering, point sound source fitting can be performed on the obstacle first, and the fitting sound source corresponding to the obstacle can be obtained by fitting a small number of point sound sources.

[0170] The upper limit of the point sound source used for fitting can be limited to 10, 20 or other values.

[0171] Specifically, we can first determine whether the size of the obstacle is large, such as whether one or more of the length, width, area, etc. of the obstacle is greater than the corresponding threshold. If so, obtain the geometric outline of the obstacle, and perform directivity fitting on the geometric outline of the obstacle through multiple point sound sources to obtain the fitting sound source corresponding to the obstacle to reduce the amount of calculation.

[0172] After obtaining the fitting sound source corresponding to the obstacle, the fitting sound source is modeled based on the model corresponding to the model label of the obstacle to obtain the sound source model of the obstacle, thereby achieving the simplification of the sound source model of the large-size obstacle.

[0173] Optionally, based on the model corresponding to the model label, constructing a sound source model of the obstacle, including: performing spatial mapping projection on the obstacle to obtain a projection of the obstacle on the outer surface of the vehicle to obtain an obstacle projection; modeling the obstacle projection based on the model corresponding to the model label to obtain a sound source model of the obstacle.

[0174] For larger obstacles or scenes with multiple obstacles, the obstacles can be projected onto the outer surface of the vehicle by connecting multiple points on the obstacle with the listening position, thereby greatly reducing the overall size of the obstacle and simplifying the computational effort of obstacle sound source modeling and rendering.

[0175] Figure 7 A schematic diagram of obstacle space mapping projection provided in an embodiment of the present application, such as Figure 7 As shown, the driver's head in the vehicle is taken as the listening position, and the obstacle is projected to the listening position by spatial mapping projection, so as to obtain the part of the obstacle projection located on the outer surface of the vehicle, and obtain the obstacle projection of the obstacle.

[0176] When rendering the sound source model constructed based on obstacle projection, information such as the distance between the obstacle and the vehicle and the shape of the obstacle can be retained to improve the authenticity of the rendering.

[0177] After obtaining the sound source model of the obstacle and the prompt content, spatial encoding is required. Based on algorithms such as Green's function, spherical harmonic domain transformation, and Kirchhoff integral, the prompt content with the sound source model of the obstacle as the sound source can be spatially encoded to obtain the spatial encoding and metadata corresponding to the prompt sound. When rendering, the spatial encoding and metadata corresponding to the prompt sound are first spatially decoded, and then the audio data corresponding to the decoded prompt sound is rendered based on the corresponding rendering algorithm to obtain the rendered prompt sound, and the rendered prompt sound is played through the vehicle's playback system, i.e., the speaker array.

[0178] Figure 8 The structure diagram of a vehicle audio system provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the vehicle audio system includes: an obstacle warning sound generating device, a sensor and a renderer. The sensors are usually multiple and are used to collect data of obstacles outside the vehicle; the obstacle warning sound generating device is used to execute the obstacle warning method provided in any embodiment of the present application to generate the obstacle warning sound; the renderer is used to render the warning sound so as to play the warning sound through the speaker array of the vehicle.

[0179] In some embodiments, the in-vehicle audio system also includes a speaker array on the vehicle.

[0180] In some embodiments, the vehicle audio system includes an ADAS, which is used to determine whether there is an obstacle based on data collected by a sensor. If so, the data corresponding to the obstacle collected by the sensor is sent to an obstacle warning sound generating device to generate a warning sound for the obstacle.

[0181] An embodiment of the present application also provides an obstacle prompt device, including: a content and label determination module, used to determine the prompt content and model label of the obstacle based on the obstacle data collected by the vehicle sensor; a sound source modeling module, used to construct a sound source model of the obstacle based on the model corresponding to the model label; based on the sound source model of the obstacle and the prompt content, generate a prompt sound of the obstacle.

[0182] In some embodiments, the content and label determination module includes: an attribute information determination unit, used to determine the attribute information of the obstacle based on the data collected by the vehicle sensor, the attribute information is used to describe at least one of the type, size, material, temperature, moving speed, position relative to the vehicle, distance from the vehicle and sound absorption coefficient of the obstacle; a content and label determination unit, used to obtain audio data collected by a microphone, and determine the prompt content and model label of the obstacle based on the audio data and the attribute information.

[0183] In some embodiments, the content and label determination unit is specifically used to: if the obstacle is a silent object, determine the prompt content and model label of the obstacle based on the attribute information; if the obstacle is a sound-emitting object, obtain the audio data collected by the microphone, and determine whether the audio data includes the audio data corresponding to the obstacle; if so, determine the prompt content of the obstacle based on the audio data corresponding to the obstacle, and determine the model label of the obstacle based on the attribute information; if not, determine the prompt content and model label of the obstacle based on the attribute information.

[0184] In some embodiments, when the obstacle is a sound-emitting object, if the audio data includes audio data corresponding to the obstacle, the content and label determination unit is specifically used to: determine the prompt content of the obstacle based on the audio data corresponding to the obstacle; if the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is greater than a first distance, determine that the model label corresponding to the obstacle is a far-field low-speed model label; if the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is less than or equal to the first distance, determine that the model label corresponding to the obstacle is a near-field low-speed model label.

[0185] In some embodiments, when the obstacle is a sound-emitting object, if the audio data does not include audio data corresponding to the obstacle, the content and label determination unit is specifically used to: determine the prompt scene of the obstacle based on the attribute information, and based on the prompt scene, determine the prompt content of the obstacle from a plurality of pre-made standard prompt contents; and determine the model label of the obstacle based on the attribute information.

[0186] In some embodiments, when the obstacle is a silent object, the content and label determination unit is specifically used to: if the obstacle is determined to be a rapidly approaching obstacle based on the distance between the obstacle and the vehicle and the speed of the obstacle, and the distance between the obstacle and the vehicle is less than the braking safety distance corresponding to the current speed of the vehicle, then determine that the model label corresponding to the obstacle is a high-speed collision label, and the prompt content corresponding to the obstacle is a high-speed collision prompt content; if the obstacle is a traffic light, then determine that the model label corresponding to the obstacle is a traffic light label, and the prompt content corresponding to the obstacle is a traffic light prompt content; if the vehicle is in a reversing state, the obstacle is located behind the vehicle, and the distance between the obstacle and the vehicle is less than a second distance, then determine that the model label corresponding to the obstacle is a reversing label, and the prompt content corresponding to the obstacle is a reversing prompt content.

[0187] In some embodiments, the sound source modeling module is specifically used to: determine the coordinates of the obstacle in the coordinate system of the vehicle based on the data collected by the vehicle sensor; perform coordinate transformation on the obstacle, converting the coordinates of the obstacle into coordinates in a spherical coordinate system with the listening position as the origin; and construct a sound source model of the obstacle based on the transformed coordinates of the obstacle using the model corresponding to the model label.

[0188] In some embodiments, the sound source modeling module is specifically used to: fit the sound source corresponding to the obstacle based on the geometric contour of the obstacle and based on multiple point sound sources to obtain a fitted sound source; model the fitted sound source based on the model corresponding to the model label to obtain a sound source model of the obstacle.

[0189] In some embodiments, the sound source modeling module is specifically used to: perform spatial mapping projection on the obstacle to obtain the projection of the obstacle on the outer surface of the vehicle to obtain an obstacle projection; model the obstacle projection based on the model corresponding to the model label to obtain a sound source model of the obstacle.

[0190] In some embodiments, the device also includes: a rendering algorithm label determination module, which is used to determine the rendering algorithm label based on the model label, the type of the obstacle or the type of the prompt content; a prompt sound rendering module, which is used to render the prompt sound based on the rendering algorithm corresponding to the rendering algorithm label, and obtain the gain of each speaker channel of the vehicle when rendering the prompt sound, so as to play the rendered prompt sound through each speaker of the vehicle.

[0191] In some embodiments, the rendering algorithm label determination module is specifically used for: if the model label is a traffic light label or the obstacle is a traffic light, then determining that the rendering algorithm label is a traffic light rendering label, and the rendering algorithm corresponding to the traffic light rendering label is a sound bed rendering algorithm or a panoramic sound rendering algorithm; if the obstacle is a moving object, then determining that the rendering algorithm label is a moving object rendering label, and the rendering algorithm corresponding to the moving object rendering label is a translation algorithm or a wave field synthesis rendering algorithm.

[0192] Fig. 9 The present invention provides a schematic diagram of an obstacle warning system provided in an embodiment of the present invention. The obstacle warning system is arranged in a vehicle and is used for obstacle detection, sound source modeling and warning sound playback. Fig. 9 As shown, the obstacle warning system includes a sensor system, an obstacle attribute recognition module, a sound source modeling and space coding module, a rendering module and a playback system.

[0193] The sensor system includes multiple sensors installed on the vehicle, such as infrared sensors, radar sensors, image sensors, ultrasonic sensors, microphones, etc., to collect data on obstacles, including but not limited to raw data corresponding to attributes such as size, position, moving speed, temperature, audio data, etc.

[0194] The obstacle attribute recognition module is used to determine the attribute information of the obstacle, such as the category, material, size, orientation, speed, audio data, etc. of the obstacle based on the original data of the obstacle fed back by the sensor system, such as signal strength, image, distance, temperature, size, etc.

[0195] The sound source modeling and spatial coding module is used to determine the prompt content, model label and rendering algorithm label of the obstacle based on the attribute information of the obstacle, and to establish the sound source model of the obstacle based on the model label of the obstacle, and to spatially encode the prompt content using the sound source model of the obstacle as the sound source.

[0196] The rendering module is responsible for spatial decoding and rendering of stereo prompt sounds. It can perform spatial decoding on the spatial coding corresponding to the prompt sounds based on the spatial coding and metadata corresponding to the prompt sounds, and render the decoded audio data, such as objects, sound beds and metadata, based on the corresponding rendering algorithms.

[0197] The playback system includes a speaker array in the vehicle, such as a 7.1-channel speaker array, through which the prompt sound is played.

[0198] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the processor implements a method provided in any embodiment of the present application.

[0199] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method provided in any embodiment of the present application.

[0200] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0201] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An obstacle prompting method, characterized in that: include: Based on the obstacle data collected by the vehicle sensor, determine the obstacle prompt content and model label; Based on the model corresponding to the model label, construct a sound source model of the obstacle; A prompt sound of the obstacle is generated based on the sound source model of the obstacle and the prompt content.

2. The method according to claim 1, characterized in that Based on the data collected by the vehicle sensors, the obstacle prompt content and model label are determined, including: Determining attribute information of the obstacle based on data collected by the vehicle sensor, the attribute information being used to describe at least one of the type of the obstacle, the moving speed, the position relative to the vehicle, and the distance from the vehicle; Get the audio data collected by the microphone; Based on the audio data and the attribute information, the prompt content and the model label of the obstacle are determined.

3. The method according to claim 2, characterized in that Acquiring audio data collected by a microphone, and determining the prompt content and model label of the obstacle based on the audio data and the attribute information, including: If the obstacle is a silent object, determining the prompt content and model label of the obstacle based on the attribute information; If the obstacle is a sound-emitting object, then obtaining audio data collected by a microphone, and determining whether the audio data includes audio data corresponding to the obstacle; If yes, determining the prompt content of the obstacle based on the audio data corresponding to the obstacle, and determining the model label of the obstacle based on the attribute information; If not, the prompt content and model label of the obstacle are determined based on the attribute information.

4. The method according to claim 3, characterized in that Determining a model label of the obstacle based on the attribute information includes: If the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is greater than a first distance, determining that the model label corresponding to the obstacle is a far-field low-speed model label; If the category of the obstacle is a pedestrian and the distance between the obstacle and the vehicle is less than or equal to the first distance, it is determined that the model label corresponding to the obstacle is a near-field low-speed model label.

5. The method according to claim 3, characterized in that: Determining the prompt content of the obstacle based on the attribute information includes: Based on the attribute information, determining a prompt scene of the obstacle; Based on the prompt scenario, the prompt content of the obstacle is determined from a plurality of pre-made standard prompt contents.

6. The method according to claim 3, characterized in that When the obstacle is a silent object, determining a model label of the obstacle and determining a prompt content of the obstacle based on the attribute information includes: If the obstacle is determined to be a fast approaching obstacle based on the distance between the obstacle and the vehicle and the speed of the obstacle, and the distance between the obstacle and the vehicle is less than the braking safety distance corresponding to the current speed of the vehicle, then the model label corresponding to the obstacle is determined to be a high-speed collision label, and the prompt content corresponding to the obstacle is a high-speed collision prompt content; If the obstacle is a traffic light, it is determined that the model label corresponding to the obstacle is a traffic light label, and the prompt content corresponding to the obstacle is a traffic light prompt content; If the vehicle is in a reversing state, the obstacle is located behind the vehicle, and the distance between the obstacle and the vehicle is less than a second distance, it is determined that the model label corresponding to the obstacle is a reversing label, and the prompt content corresponding to the obstacle is a reversing prompt content.

7. The method according to claim 1, characterized in that Constructing a sound source model of the obstacle based on the model corresponding to the model label, including: Based on the geometric contour of the obstacle and multiple point sound sources, a sound source corresponding to the obstacle is fitted to obtain a fitting sound source; The fitting sound source is modeled based on the model corresponding to the model label to obtain a sound source model of the obstacle.

8. The method according to claim 1, characterized in that Constructing a sound source model of the obstacle based on the model corresponding to the model label, including: Performing spatial mapping projection on the obstacle to obtain a projection of the obstacle on the outer surface of the vehicle, thereby obtaining an obstacle projection; The obstacle projection is modeled based on the model corresponding to the model label to obtain a sound source model of the obstacle.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Determine a rendering algorithm label based on the model label, the type of the obstacle, or the type of the prompt content; The prompt sound is rendered based on the rendering algorithm corresponding to the rendering algorithm label, and the gain of each speaker channel of the vehicle when rendering the prompt sound is obtained, so as to play the rendered prompt sound through each speaker of the vehicle.

10. The method according to claim 9, characterized in that Determining a rendering algorithm label based on the model label, the type of the obstacle, or the type of the prompt content includes: If the model label is a traffic light label or the obstacle is a traffic light, determining that the rendering algorithm label is a traffic light rendering label, and the rendering algorithm corresponding to the traffic light rendering label is a sound bed rendering algorithm or a panoramic sound rendering algorithm; If the obstacle is a moving object, it is determined that the rendering algorithm tag is a moving object rendering tag, and the rendering algorithm corresponding to the moving object rendering tag is a translation algorithm or a wave field synthesis rendering algorithm.

11. A vehicle audio system, arranged in a vehicle, characterized in that: include: Obstacle warning sound generation equipment, sensors and renderers; The sensor is used to collect data of obstacles outside the vehicle; The obstacle warning sound generating device is used to execute the method provided in any one of claims 1 to 10 to generate the obstacle warning sound; The renderer is used for rendering the prompt sound so as to play the prompt sound through a speaker array of the vehicle.

Images