Methods, computer program products, controllers, and vehicles for analyzing ambient sounds outside the vehicle.
By using large language models and multimodal acoustic detection algorithms, combined with vehicle environmental status information, the system identifies and analyzes external risk sounds, generates reasonable driving instructions and directional sound alerts, and solves the limitations and defects of existing technologies in external environmental sound analysis, achieving more accurate risk identification and driving assistance.
Patent Information
- Application Number
- CN202611235613.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing driver assistance systems have limitations and defects in recognizing ambient sounds outside the vehicle, making it difficult to perform comprehensive and accurate analysis.
By employing a large language model combined with multimodal acoustic detection algorithms, the system acquires information about the vehicle's external environment and environmental conditions, identifies risky external sounds, determines the location of the sound source, and generates concise and crucial sound information for generating autonomous driving or assisted driving commands and directional sound alerts.
It enables a more comprehensive and accurate analysis of ambient sounds outside the vehicle, generates reasonable driving instructions, avoids unnecessary interference with the driver, and can prioritize critical risk sounds, providing targeted sound alerts.
Smart Images

Figure CN123104656A_ABST
Abstract
Description
Technical Field
[0001] This application relates to a method for analyzing ambient sounds outside a vehicle, a computer program product, a controller, and a vehicle. Background Technology
[0002] Vehicle driver assistance systems can greatly enhance the user experience. Some current driver assistance systems can identify potential hazards outside the vehicle by comparing acoustic signals and then warn the driver through the vehicle's audio system.
[0003] However, such a solution has limitations and flaws in many aspects. Summary of the Invention
[0004] The purpose of this application is to provide a method for analyzing the external environmental sounds of a vehicle, and in particular, to enable a more comprehensive and accurate analysis of the external environmental sounds based on a large language model.
[0005] According to a first aspect of this application, a method for analyzing the external ambient sound of a vehicle is provided, the method comprising: acquiring first input data and second input data, the first input data describing the current external ambient sound of the vehicle, and the second input data describing environmental state information suitable for assisting in the analysis of the current external ambient sound; and performing analysis on the current external ambient sound by a large language model based on the first input data and the second input data.
[0006] Through the above methods, the following technical effects are achieved in particular: the large language model can understand a broader context, thereby making a more comprehensive and accurate analysis of the sounds in the vehicle's external environment.
[0007] According to an optional embodiment of this application, the method includes: The system acquires raw data of the current external ambient sound of the vehicle, and uses a multimodal acoustic detection algorithm to identify predefined external risk sounds from the raw data and determine the location of the sound source of the external risk sounds. The first input data is generated based on the external risk sounds and the determined sound source location of the external risk sounds, and the first input data describes the external risk sounds identified in the current external environment sounds.
[0008] Through the above methods, the following technical effects are achieved: the multimodal acoustic detection algorithm can filter out predefined external risk sounds and determine their source location, thereby inputting concise and key sound information into the large language model.
[0009] According to an optional embodiment of this application, the method includes: generating autonomous driving or assisted driving instructions and / or generating reminder instructions based on the analysis, the reminder instructions being used to prompt the directional speakers on the driver's headrest of the vehicle to issue directional sound reminders to the driver of the vehicle.
[0010] Through the above methods, the following technical effects are achieved in particular: the ability to generate very reasonable autonomous driving or assisted driving commands; and the ability to issue directional sound reminders, especially in a way that does not disturb other occupants in the vehicle.
[0011] According to an optional embodiment of this application, the analysis includes: confirming whether a risk event exists based on the first input data and the second input data; and generating output text data indicating the corresponding countermeasures if a risk event exists.
[0012] The above methods achieve the following technical effect: non-risk events can be eliminated to avoid unnecessary interference with the driver.
[0013] According to an optional embodiment of this application, the analysis includes: when multiple external risk sounds are identified in the current external ambient sound, performing a priority analysis on the multiple external risk sounds in conjunction with the second input data, and prioritizing the external risk sounds with higher priority among the multiple external risk sounds.
[0014] The above methods are particularly effective in prioritizing critical incidents.
[0015] According to an optional embodiment of this application, the method includes: generating a reminder instruction, the reminder instruction being used to prompt the vehicle to play the highest priority external risk sound among the plurality of external risk sounds and / or play a semantic reminder for the highest priority external risk sound among the plurality of external risk sounds.
[0016] This effectively alerts drivers to the most critical external risks in the current environment.
[0017] According to an optional embodiment of this application, the external risk sounds include at least one of the following: emergency vehicle sirens; pedestrian shouts; children playing; animal calls.
[0018] These types of external hazard sounds basically cover the more important external hazard sounds during normal driving.
[0019] According to an optional embodiment of this application, the environmental state information includes at least one of the following: the location of the vehicle; the speed of the vehicle; the current time; and the traffic conditions around the vehicle.
[0020] This environmental information can be helpful in analyzing the sound outside the vehicle.
[0021] According to a second aspect of this application, a computer program product is provided, the computer program product comprising computer program instructions, wherein, when the computer program instructions are executed by one or more processors, the processors are capable of performing the aforementioned method for analyzing ambient sounds outside a vehicle.
[0022] The computer program product accordingly incorporates the advantages of the method.
[0023] According to a third aspect of this application, a controller is provided, the controller including a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, enable the processor to perform the aforementioned method for analyzing ambient sounds outside a vehicle.
[0024] The controller accordingly incorporates the advantages of the method.
[0025] According to a fourth aspect of this application, a vehicle is provided, the vehicle including the aforementioned controller.
[0026] The vehicle accordingly incorporates the advantages of the method. Attached Figure Description
[0027] The principles, features, and advantages of this application will be better understood below with reference to the accompanying drawings. The drawings include: Figure 1 The illustration shows a first example of the process of the method for analyzing ambient sound outside a vehicle according to this application; Figure 2 The illustration shows a second example of the process of the method for analyzing ambient sounds outside a vehicle according to this application; Figure 3 An example of hardware associated with the method is illustrated schematically; Figure 4 An example of a data flow associated with the method is illustrated schematically; Figure 5 This illustration shows an example of the environment in which the vehicle is located; Figure 6 This illustration shows another example of the environment in which the vehicle is located.
[0028] List of reference numerals 2 controllers; 30 External microphone array, 31 Other sensing devices, 32 CAN bus; 40. Driver headrest directional speakers; 41. Cloud-based audio system. 50 multimodal acoustic detection modules, 51 first data preprocessor, 52 second data preprocessor, 53 large language model; 60 Raw data of external ambient sound, 61 External risk sound data, 62 Raw data of environmental status information, 63 First input data, 64 Second input data, 65 Output text data; 70 vehicles, 71 children, 72 emergency vehicles, 720 ambulances, 73 pedestrians. Detailed Implementation
[0029] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.
[0030] Figure 1 The illustration shows a first example of the process of the method for analyzing ambient sounds outside a vehicle 70 according to this application.
[0031] Figure 2 The illustration shows a second example of the process of the method for analyzing ambient sounds outside a vehicle 70 according to this application.
[0032] Figure 3 An example of hardware associated with the method is illustrated schematically.
[0033] Figure 4 An example of a data flow associated with the method is illustrated schematically.
[0034] Figure 5 The illustration shows an example of the environment in which vehicle 70 is located.
[0035] Figure 6 This illustration shows another example of the environment in which vehicle 70 is located.
[0036] like Figure 1 and Figure 4 As shown, the method includes: Step 102: Obtain first input data 63 and second input data 64. The first input data 63 describes the current external ambient sound of the vehicle 70, and the second input data 64 describes environmental state information suitable for auxiliary analysis of the current external ambient sound. Step 104: The large language model 53 analyzes the current external ambient sound based on the first input data 63 and the second input data 64.
[0037] Through the above methods, the following technical effects are achieved: In particular, compared with traditional neural network models, the large language model 53 can understand a broader context, thereby making a more comprehensive and accurate analysis of the sounds in the vehicle's external environment.
[0038] In particular, to meet the target response time of less than 300 milliseconds, the entire process of the method can be executed in the controller 2 of the vehicle 70. The same applies to the other steps described below.
[0039] The controller 2 may include a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, enable the processor to perform the method.
[0040] It is conceivable that: - The method is executed by a controller 2 of the vehicle 70, which may be a driving domain controller or a cockpit domain controller, or an integrated central controller; or, - The method is jointly executed by multiple controllers 2 of the vehicle 70, which may include, for example, a driving domain controller and a cockpit domain controller.
[0041] However, it is also theoretically possible that the method is executed by the cloud 41; or that the method is executed collaboratively by the controller 2 of the vehicle 70 and the cloud 41.
[0042] According to an exemplary embodiment of this application, see Figure 2 and Figure 4 The method includes: Step 100: In step 100, the original external environmental sound data 60 of the current external environmental sound of the vehicle 70 is obtained, and a predefined external risk sound is identified from the original external environmental sound data 60 and the location of the sound source of the external risk sound is determined by a multimodal acoustic detection algorithm. Step 101: In step 101, the first input data 63 is generated based on the external risk sound and the determined sound source location of the external risk sound. The first input data 63 describes the external risk sound identified in the current external environment sound.
[0043] Through the above methods, the following technical effects are achieved: the multimodal acoustic detection algorithm can filter out predefined external risk sounds and determine their source location, thereby inputting concise and key sound information into the large language model 53.
[0044] Multimodal acoustic detection algorithms may involve, for example, convolutional neural network models.
[0045] For example, see Figure 3and Figure 4 In relation to hardware and data flow, this may involve, for example, the following processes: - The external microphone array 30 receives ambient sounds from outside the vehicle, and can additionally determine the location of the sound source. The external microphone array 30 also employs noise suppression technology, for example, to ensure clear capture of key sounds even under high wind speeds or traffic noise. - The external microphone array 30 sends the raw external ambient sound data 60 to the controller 2. The multimodal acoustic detection module 50 in the controller 2 uses a multimodal acoustic detection algorithm to identify whether there are predefined external risk sounds in the raw external ambient sound data 60 and to determine the location of the sound source of the identified external risk sounds. Based on this, external risk sound data 61 can be generated. - The first data preprocessor 51 in the controller 2 converts the external risk sound data 61 into first input data 63, especially first input text data, which can be understood by the large language model 53. The first input text data may involve structured text or natural language text. The first input data 63 is transmitted from the first data preprocessor 51 to the large language model 53 in the controller 2. - The controller 2 obtains environmental state information (i.e., contextual information associated with external risk sounds) suitable for assisting in the analysis of the current external ambient sound from the CAN line 32 of the vehicle 70 and / or other sensing devices 31 of the vehicle 70 (e.g., visual sensors and / or radar sensors). - The second data preprocessor 52 in the controller 2 converts the raw environmental state information data 62 into second input data 64, especially second input text data, which can be understood by the large language model 53. The second input text data can involve structured text or natural language text. The second input data 64 is transmitted from the second data preprocessor 52 to the large language model 53 in the controller 2. - The large language model 53 in controller 2 analyzes external risk sounds based on the first input data 63 and the second input data 64.
[0046] According to an exemplary embodiment of this application, see Figure 5 and Figure 6 The external risk sounds include at least one of the following: The siren of emergency vehicle number 72 sounded; The shouts of pedestrian 73; The sounds of children playing; Animal sounds.
[0047] These types of external hazard sounds basically cover the more important external hazard sounds during normal driving.
[0048] Especially when the shouts of pedestrian 73 are detected, the external risk sound data 61 is also input into the speech recognition module in the multimodal acoustic detection module 50. The speech recognition module can extract features from the pedestrian's shouts and generate corresponding text. The generated text can be added when generating the first input data 63.
[0049] According to an exemplary embodiment of this application, the environmental state information includes at least one of the following: The location of the vehicle 70 (especially its GPS location); The vehicle's speed was 70. Current time; Traffic conditions around the vehicle 70.
[0050] This environmental information can be helpful in analyzing the sound outside the vehicle.
[0051] According to an exemplary embodiment of this application, see Figure 4 The analysis includes: confirming whether a risk event exists based on the first input data 63 and the second input data 64; and generating output text data 65 indicating the corresponding countermeasures if a risk event exists.
[0052] The above methods achieve the following technical effect: non-risk events can be eliminated to avoid unnecessary interference with the driver.
[0053] According to an exemplary embodiment of this application, see Figure 2 and Figure 3 The method includes step 106, in which, based on the analysis, an autonomous driving or assisted driving instruction is generated and / or a reminder instruction is generated, the reminder instruction being used to prompt the driver headrest directional speaker 40 of the vehicle 70 to issue a directional sound reminder to the driver of the vehicle 70.
[0054] Through the above methods, the following technical effects are achieved in particular: the ability to generate very reasonable autonomous driving or assisted driving commands; and the ability to issue directional sound reminders, especially in a way that does not disturb other occupants in the vehicle.
[0055] According to an exemplary embodiment of this application, the analysis includes: when multiple external risk sounds are identified in the current external ambient sound, performing a priority analysis on the multiple external risk sounds in conjunction with the second input data 64, and prioritizing the external risk sounds with higher priority among the multiple external risk sounds.
[0056] The above methods are particularly effective in prioritizing critical incidents.
[0057] For example, step 106 includes: generating a reminder instruction, the reminder instruction being used to prompt the vehicle 70 to play the highest priority external risk sound among the plurality of external risk sounds and / or play a semantic reminder for the highest priority external risk sound among the plurality of external risk sounds.
[0058] This effectively alerts drivers to the most critical external risks in the current environment.
[0059] In the case of prioritizing the multiple external risk sounds mentioned above, especially when it is confirmed that there is a risk event related to the highest priority external risk sound, an alert command is generated for the highest priority external risk sound.
[0060] The following is Figure 5 Here is an example illustrating the operating process of vehicle 70. Figure 5 In this scenario, vehicle 70 is traveling on a highway. For example, vehicle 70 may operate according to the following procedure: -The vehicle's external microphone array 30 receives ambient sounds from outside the vehicle; - The multimodal acoustic detection algorithm identifies the siren of ambulance 720 from the ambient sounds outside the vehicle, and the siren of ambulance 720 is coming from behind vehicle 70; - The first input data 63 generated accordingly is, for example: There is an ambulance siren to the left rear; - The second input data 64 generated based on the original environmental state information data 62 is, for example: the vehicle is on the highway and the ambulance is on the same road as the vehicle. -On the highway, the siren of ambulance 720 has the highest priority by default, and the analysis of the large language model 53 shows that there is a related risk event. - The large language model 53 generates output text data 65: reminding the driver "An ambulance is approaching from the left rear, please keep to the right"; - The data processor generates corresponding reminder instructions based on the output text data 65; - The cockpit domain controller, based on the reminder command, drives the driver's headrest directional speaker 40 to play a semantic reminder that "an ambulance is approaching from the left rear, please keep to the right" and / or play the ambulance siren 720.
[0061] Unlike the above, if the second input data 64 generated based on the original environmental state information data 62 is: the vehicle is on the highway and the ambulance is on the opposite road, then the large language model 53 confirms that there is no risk event.
[0062] The following is Figure 6 Here is an example illustrating the operating process of vehicle 70. Figure 6 In the middle, vehicle 70 passes by the school gate. For example, vehicle 70 can operate according to the following procedure: -The vehicle's external microphone array 30 receives ambient sounds from outside the vehicle; - The multimodal acoustic detection algorithm identifies the sounds of children playing to the right front and pedestrians shouting to the front from the outside of the vehicle. - The first input data 63 generated accordingly is, for example: There are children playing to the right front, and pedestrians shouting "Watch out for cars" directly in front; - The second input data 64 generated based on the original environmental state information data 62 is, for example: the vehicle is on the road in front of the school, it is the time of school dismissal, and the vehicle speed is 10 kilometers per hour; - Analysis of the large language model 53: In the current environment, the sounds of children playing 71 have high priority, and there is a risk of collision with children 71; - The large language model 53 generates output text data 65: reminding the driver "There is a child in front of you on the right, drive carefully and pay attention to controlling your speed"; - The data processor generates corresponding reminder instructions based on the output text data 65; - The cockpit domain controller, based on reminder commands, drives the driver's headrest directional speakers to play a semantic reminder: "There is a child in front of you on the right. Drive carefully and control your speed."
[0063] For risk events, drivers can provide feedback. Imagine a pop-up window for a risk event appearing on the central control screen, allowing the driver to manually confirm the event or choose to ignore it.
[0064] For example, a driver-customizable settings function can also be provided. The driver can adjust the alert parameters, for instance, via the central control screen. - Drivers can enable or disable alerts for specific sound types, such as turning off alerts for children playing sounds on highways; - Drivers can set the alert volume level and alert sensitivity (high, medium, low) to suit different driving environments.
[0065] According to an exemplary embodiment of this application, such as Figure 2 As shown, the method includes step 108, in which risk events and driver feedback are recorded and uploaded to the cloud 41. The cloud can then use this data for subsequent model optimization (e.g., updating the multimodal acoustic detection algorithm or the corresponding convolutional neural network model through machine learning) and to generate a driving safety report.
[0066] The system can periodically upload anonymized sound detection records, priority analysis results, and driver interaction data to the cloud. The upload process uses an encryption protocol to ensure data privacy and security.
[0067] Cloud 41 can perform aggregated analysis based on historical data (such as audio event statistics for the past 7 or 30 days) for the following purposes: - Optimize the context analysis model of the large language model 53 (e.g., adjust the sound priority rules according to regional traffic patterns). - Generate regionally specific sound heat maps to help predict high-risk areas (such as areas around schools during dismissal time).
[0068] Furthermore, it is conceivable that drivers can view driving safety reports through an app on the vehicle's 70, summarizing their responses to key external risk sounds during driving and providing suggestions for improvement.
[0069] In this context, environmental state information is specifically understood to encompass both the vehicle's environmental information and its state information.
[0070] This specification provides method steps as described in the embodiments or flowcharts, but may include more or fewer steps based on conventional or non-inventive labor.
[0071] In the accompanying drawings and specific embodiments, the method steps are presented in a particular order. However, this should be understood as merely an example and not a mandatory requirement. Where feasible in principle, the order of these method steps may be arbitrarily changed or may be performed with temporal overlap without exceeding the scope of protection of this application.
[0072] Although specific embodiments of this application are described in detail herein, they are given for illustrative purposes only and should not be construed as limiting the scope of this application. Various substitutions, modifications, and alterations can be conceived without departing from the spirit and scope of this application.
Claims
1. A method for analyzing an off-vehicle environmental sound of a vehicle, characterized by, The method includes: Acquire first input data (63) and second input data (64), wherein the first input data (63) describes the current external ambient sound of the vehicle (70), and the second input data (64) describes environmental state information suitable for auxiliary analysis of the current external ambient sound; The large language model (53) analyzes the current external ambient sound based on the first input data (63) and the second input data (64).
2. The method for analyzing a vehicle's off-board environmental sound according to claim 1, characterized by, The method includes: The vehicle (70) acquires the original data (60) of the current external environment sound of the vehicle, and identifies predefined external risk sounds from the original data (60) of the external environment sound using a multimodal acoustic detection algorithm and determines the sound source location of the external risk sounds. The first input data (63) is generated based on the external risk sounds and the determined sound source location of the external risk sounds. The first input data (63) describes the external risk sounds identified in the current external environment sounds.
3. The method for analyzing an off-board sound of a vehicle according to claim 1 or 2, characterized by, The method includes: generating autonomous driving or assisted driving instructions and / or generating reminder instructions based on the analysis, the reminder instructions being used to prompt the driver headrest directional speaker (40) of the vehicle (70) to issue a directional sound reminder to the driver of the vehicle (70).
4. The method for analyzing ambient sound outside a vehicle according to claim 1 or 2, characterized in that, The analysis includes: confirming whether a risk event exists based on the first input data (63) and the second input data (64), and generating output text data (65) indicating the corresponding countermeasures if a risk event exists.
5. The method for analyzing ambient sound outside a vehicle according to claim 2, characterized in that, The analysis includes: when multiple external risk sounds are identified in the current external environment sound, the multiple external risk sounds are prioritized in combination with the second input data (64), and the external risk sounds with high priority among the multiple external risk sounds are processed first. The method includes: generating a reminder instruction, the reminder instruction being used to prompt the vehicle (70) to play the highest priority external risk sound among the plurality of external risk sounds and / or play a semantic reminder for the highest priority external risk sound among the plurality of external risk sounds.
6. The method for analyzing ambient sound outside a vehicle according to claim 2, characterized in that, The external risk sounds include at least one of the following: The siren of an emergency vehicle (72); The shouts of pedestrians (73); The sounds of children (71) playing; Animal sounds.
7. The method for analyzing ambient sound outside a vehicle according to claim 1 or 2, characterized in that, The environmental status information includes at least one of the following: The position of the vehicle (70); The speed of the vehicle (70); Current time; Traffic conditions around the vehicle (70).
8. A computer program product, characterized in that, The computer program product includes computer program instructions, wherein when the computer program instructions are executed by one or more processors, the processors are capable of performing a method for analyzing ambient sounds outside a vehicle according to any one of claims 1 to 7.
9. A controller (2), characterized in that, The controller (2) includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, enable the processor to perform a method for analyzing ambient sounds outside the vehicle according to any one of claims 1 to 7.
10. A vehicle (70), characterized in that, The vehicle (70) includes the controller (2) according to claim 9.
