A vehicle-mounted sensing system and a data processing method

Abstract

The present invention discloses a vehicle-mounted sensing system, which includes a first sensing device, a second sensing device and a central processing device. The first sensing device is used to obtain environmental data around the vehicle and determine obstacle information around the vehicle based on the environmental data. The obstacle information includes the size, coordinates and motion state of the obstacle. The second sensing device is used to obtain audio data around the vehicle and determine sound source information around the vehicle based on the audio data. The sound source information includes the acoustic coordinates and acoustic size of the sound source. The central processing device is electrically connected to the first sensing device and receives the obstacle information determined by the first sensing device. The central processing device is electrically connected to the second sensing device and receives the sound source information determined by the second sensing device. The central processing device processes the received obstacle information and sound source information and identifies obstacles around the vehicle and / or determines the environmental wind speed of the vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle environment perception devices, and in particular, to an in-vehicle perception system and a data processing method. Background Art

[0002] In recent years, with the continuous development of automotive electronic technologies such as sensors, intelligent driving vehicles, as a solution to improve vehicle driving safety, reduce traffic accidents and traffic congestion, have rapidly become an important future development direction in the automotive industry. Generally, the system of an intelligent driving vehicle can be divided into several modules: perception, decision-making, prediction, planning, and control. Among them, the perception module is responsible for detecting the relevant environment during the driving process of the intelligent driving vehicle. The relevant environment specifically includes: traffic participants around the intelligent driving vehicle, lane lines, road signs, traffic lights, etc.

[0003] During the driving process of an intelligent driving vehicle, whether the perception module can effectively, accurately, and timely detect the relevant environment during the driving process of the intelligent driving vehicle directly determines the safety of the intelligent driving vehicle during the driving process. Therefore, for an intelligent driving vehicle, an effective, accurate, and timely perception module is very important.

[0004] Currently, in the field of intelligent driving, perception systems generally perceive the external environment based on sensors such as cameras, lidar, millimeter-wave radars, and ultrasonic waves. In the face of a complex and changeable traffic environment, traditional perception methods have deficiencies. For example, special vehicles such as fire trucks, ambulances, and police cars will sound alarms during driving in order to pass quickly. At this time, the intelligent driving vehicle should timely plan and execute avoidance actions according to the alarm sound. However, traditional perception cannot recognize the alarm sound and perform spatial positioning on the sound source; surrounding vehicles make sharp tire noises or relatively large engine noises due to emergency braking, rapid acceleration, or large lateral maneuvers. At this time, the autonomous driving vehicle should identify the position and driving intention of the target vehicle according to the sharp tire noise or relatively large engine noise and appropriately avoid it. However, traditional perception methods cannot recognize the sharp tire noise or relatively large engine noise and cannot identify the position and driving intention of the target vehicle according to acoustic information; when the target around the vehicle is completely or partially blocked, the perception frame obtained by traditional perception will disappear or the size will be distorted. At this time, referring to the engine noise and tire noise on the road can be used to improve the recognition accuracy of the target around the vehicle. However, traditional perception cannot recognize the tire noise and engine noise.

[0005] In summary, the sound signals on the road are of great value to the safe driving of intelligent driving vehicles. However, currently in the field of intelligent driving vehicles, traditional perception systems only use sensors such as cameras, millimeter-wave radars, and lidar, and do not have sensors with the function of detecting sound signals, which makes intelligent driving vehicles unable to recognize the sound signals around the vehicle. Summary of the Invention

[0006] To solve at least one aspect of the above problems, the present invention provides a vehicle-mounted sensing system, including: a first sensing device configured to acquire environmental data around the vehicle and determine obstacle information around the vehicle based on the environmental data, where the obstacle information includes the size, coordinates, and motion state of the obstacle; a second sensing device configured to acquire audio data around the vehicle and determine sound source information around the vehicle based on the audio data, where the sound source information includes the acoustic coordinates and acoustic size of the sound source; and a central processing device electrically connected to the first sensing device and receiving the obstacle information determined by the first sensing device, the central processing device being electrically connected to the second sensing device and receiving the sound source information determined by the second sensing device, the central processing device processing the received obstacle information and sound source information to identify obstacles around the vehicle and / or determine the environmental wind speed of the vehicle.

[0007] Preferably, the second sensing device includes: at least one microphone array fixedly arranged on the vehicle for acquiring audio data around the vehicle; an acoustic electronic control unit arranged inside the vehicle, the acoustic electronic control unit being electrically connected to the at least one microphone array and electrically connected to the central processing device, the acoustic electronic control unit being configured to receive and process the audio data acquired by the at least one microphone array to determine the sound source information and send the sound source information to the central processing device.

[0008] Preferably, the at least one microphone array is arranged at any one position or a combination of multiple positions among the body sheet metal between the B-pillar and the roof inside the vehicle, the body sheet metal between the B-pillar and the floor inside the vehicle, the body sheet metal between the A-pillar and the roof inside the vehicle, the body sheet metal between the A-pillar and the floor inside the vehicle, the body sheet metal between the C-pillar and the roof inside the vehicle, the body sheet metal at the position between the C-pillar and the floor inside the vehicle, and under the front row seats.

[0009] Preferably, the acoustic electronic control unit is fixedly arranged on the armrest sheet metal inside the vehicle.

[0010] Preferably, the first sensing device includes: a millimeter-wave radar module fixedly arranged on the vehicle for obtaining environmental data of a preset area outside the vehicle; a camera module fixedly arranged on the vehicle for obtaining image data around the vehicle; a first processing unit electrically connected to the millimeter-wave radar module and the camera module, the first processing unit being configured to receive and process the environmental data and the image data to determine obstacle information around the vehicle, and send the obstacle information to the central processing device.

[0011] Preferably, the millimeter-wave radar module is arranged at any one or a combination of multiple positions including the midpoint position of the front bumper of the vehicle, the two end positions of the front bumper of the vehicle, the midpoint position of the rear bumper of the vehicle, the left side body position of the vehicle, and the right side body position of the vehicle.

[0012] Preferably, the millimeter-wave radar module includes a millimeter-wave radar control unit and at least one millimeter-wave radar, the millimeter-wave radar control unit being electrically connected to the at least one millimeter-wave radar, and the millimeter-wave radar control unit receiving and processing signals of the millimeter-wave radar.

[0013] Preferably, the camera module is arranged at any one or a combination of multiple positions including the midpoint position of the radiator grille, the position of the right rearview mirror housing, the position of the left rearview mirror housing, and the midpoint position of the rear bumper.

[0014] Preferably, the camera module includes a camera control unit and at least one camera, the camera control unit being electrically connected to the at least one camera, and the camera control unit receiving and processing signals of the camera.

[0015] Preferably, the first sensing device further includes a lidar module electrically connected to the first processing unit, the lidar module being configured to obtain environmental data around the vehicle and send it to the first processing unit.

[0016] On the other hand, the present invention provides a data processing method using the vehicle-mounted sensing system as described above, including: obtaining environmental data around the vehicle by using a first sensing device, and determining obstacle information around the vehicle based on the environmental data, where the obstacle information includes the size, coordinates, and motion state of the obstacle; obtaining audio data around the vehicle by using a second sensing device, and determining sound source information around the vehicle based on the audio data, where the sound source information includes the acoustic coordinates and acoustic size of the sound source; electrically connecting a central processing unit to the first sensing device and receiving the obstacle information determined by the first sensing device, electrically connecting the central processing unit to the second sensing device and receiving the sound source information determined by the second sensing device, and the central processing unit processes the obstacle information and the sound source information and identifies obstacles around the vehicle and / or determines the environmental wind speed of the vehicle.

[0017] Preferably, the step of obtaining audio data around the vehicle by using a second sensing device and determining sound source information around the vehicle based on the audio data includes: collecting an environmental audio data set by using at least one microphone array, where the environmental audio data set includes audio files of various vehicles making sounds under various driving conditions, various road conditions, and various weather environments; determining a sound source feature recognition algorithm by using an acoustic electronic control unit according to the environmental audio data set; collecting audio data around the vehicle by using the at least one microphone array, and the acoustic electronic control unit receiving the audio data around the vehicle and identifying sound source features based on the sound source feature recognition algorithm, where the sound source features include the components, categories, and conditions of the sound-emitting parts; the acoustic electronic control unit determining the acoustic size of the sound source according to the sound source features, and determining the acoustic coordinates of the sound source based on a sound source localization algorithm according to the audio data around the vehicle.

[0018] Preferably, the components of the sound source features include any one or a combination of several of tire noise, engine noise, exhaust noise, rearview mirror wind noise, and in-vehicle speaker audio, and the in-vehicle speaker audio includes alarms emitted by ambulances, fire trucks, and police cars.

[0019] Preferably, the categories of the sound source features include passenger cars, commercial buses, trucks, ambulances, fire trucks, police cars, and the sound source cannot be identified.

[0020] Preferably, the conditions of the sound source features include emergency braking, rapid acceleration, and steady driving.

[0021] Preferably, the sound source information further includes an acoustic frame of the sound source, where the acoustic frame is the spatial distribution of the acoustic size of the sound source.

[0022] Preferably, the step of the acoustic electronic control unit determining the acoustic frame of the sound source includes: determining the current acoustic coordinates of the sound source and the acoustic coordinates at the previous moment; taking the direction determined by connecting the current acoustic coordinates of the sound source and the acoustic coordinates at the previous moment as the length direction of the acoustic frame, taking the current acoustic coordinates of the sound source as the center of the acoustic frame, and taking the length and width of the acoustic size of the sound source as the length and width of the acoustic frame.

[0023] Preferably, when the sound source is the whole vehicle, the step of the acoustic electronic control unit determining the acoustic frame includes: when the relative distance change between the center coordinates of the acoustic frames of multiple sound sources is less than a set first threshold, determining that the multiple sound sources belong to the same vehicle; taking the smallest rectangular frame including the acoustic frames of multiple sound sources belonging to the same vehicle as the acoustic frame of the whole vehicle.

[0024] Preferably, the sound source information further includes the center coordinates of the wind speed sound source, the wind speed sound source frequency, and the wind speed sound source intensity.

[0025] Preferably, the step of determining the sound source information of the wind speed sound source includes: using at least one microphone array to collect a wind noise sound data set, where the wind noise sound data set includes audio files of air vortex sounds when the wind blows through different static obstacles at different wind speeds; using the acoustic electronic control unit to determine a wind speed feature recognition algorithm according to the wind noise sound data set; using the at least one microphone array to collect audio data around the vehicle, and using the acoustic electronic control unit to receive the audio data around the vehicle and judge whether it is a wind speed sound source based on the wind speed feature recognition algorithm; when it is judged that the audio data around the vehicle collected by the at least one microphone array is a wind speed sound source, the acoustic electronic control unit determines the acoustic information of the wind speed sound source based on the audio data around the vehicle and based on the sound source localization algorithm.

[0026] Preferably, the step of using the first sensing device to determine the obstacle information around the vehicle includes: using the millimeter wave radar module and the camera module to obtain environmental data, and identifying static obstacles and dynamic obstacles based on the environmental data; using the first processing unit to fuse the recognition results of the millimeter wave radar module and the camera module to determine the size, coordinates, and motion state of the obstacles around the vehicle.

[0027] Preferably, the step of using the first sensing device to obtain the obstacle information around the vehicle includes: using the millimeter wave radar module, the camera module, and the lidar module to obtain environmental data, and identifying static obstacles and dynamic obstacles based on the environmental data; using the first processing unit to fuse the recognition results of the millimeter wave radar module, the camera module, and the lidar module to determine the size, coordinates, and motion state of the obstacles around the vehicle.

[0028] Preferably, when the central processing unit determines that the distance between the acoustic coordinates of the sound source and the center coordinates of the obstacle is less than a set second threshold, the obstacle and the sound source are the same obstacle, and the spatial distribution of the geometric size of the obstacle is determined according to the minimum rectangular frame including the acoustic frame of the sound source and the obstacle perception frame.

[0029] Preferably, the steps for the central processing unit to determine the ambient wind speed include: using the second sensing device to determine the center coordinates of the wind speed sound source, using the first sensing device to determine the coordinates of the static obstacle, taking the center of the wind speed sound source and the static obstacle as a wind speed measurement pair, and determining the distance between the wind speed measurement pairs according to the center coordinates of the wind speed sound source and the coordinates of the static obstacle: Determine the angle between the line connecting the wind speed measurement pairs and the vehicle driving direction: where x NWM and y NWM are the coordinates of the center of the wind speed sound source in the geodetic coordinate system, x SON and y SON are the coordinates of the static obstacle in the geodetic coordinate system; calculate the average value of the angles between multiple wind speed measurement pairs and the vehicle driving direction Set the angle threshold Δθ WANM , distance threshold MaxD NM , and take the wind speed measurement pairs that satisfy D NM <MaxD NM and as the counting measurement points; calculate the wind noise frequency of the counting measurement points: f NWi is the frequency at which the audio emitted by the center of the wind speed sound source corresponding to the i-th counting measurement point is received, u is the speed of sound, v is the vehicle speed, and α i is the angle between the line connecting the center of the wind speed sound source of the i-th counting measurement point and the vehicle and the vehicle movement direction; determine the ambient wind speed of the vehicle according to the determined wind noise frequencies of multiple counting measurement points: where, where, the value range of i corresponds to the total number n of the counting measurement points, dB NWi is the sound intensity corresponding to the i-th counting measurement point, and v WSi is the wind speed of the i-th counting measurement point.

[0030] Preferably, the direction of the ambient wind speed of the vehicle is: where, where D Wi is the distance between the center of the wind speed sound source corresponding to the i-th counting measurement point and the static obstacle, n is the total number of the counting measurement points, and θ WAi is the angle between the line connecting the center of the wind speed sound source and the static obstacle in the i-th counting measurement point and the ground.

[0031] Preferably, the component of the environmental wind speed in the vehicle traveling direction is: v WSo = v WS × cos(θ WA - θ Ego ) - v Ego , and the component in the vertical direction is: v WSa = v WS × sin(θ WA - θ Ego ), where θ Ego is the yaw angle of the vehicle, and v Ego is the speed of the vehicle.

[0032] Preferably, the central processing unit processes the obstacle information and the sound source information to determine the danger coefficient of the obstacle: K Nt = K NC × K NT , where K NC is the acoustic condition danger coefficient, and K NT is the obstacle type danger coefficient.

[0033] Preferably, the acoustic condition danger coefficient includes an emergency acceleration coefficient, and the emergency acceleration coefficient is determined by the following formula: K acc = K accN × f accN , K accN is the set emergency acceleration proportionality coefficient, and f accN is the emergency acceleration tire noise frequency determined by the second sensing device.

[0034] Preferably, the acoustic condition danger coefficient includes an emergency braking coefficient, and the emergency braking coefficient is determined by the following formula: K bra = K braN × f braN , K braN is the set emergency braking proportionality coefficient, and f braN is the emergency braking tire noise frequency determined by the second sensing device.

[0035] Preferably, the acoustic condition danger coefficient includes a steady driving coefficient, and the steady driving coefficient is equal to the set steady driving proportionality coefficient.

[0036] The on-vehicle sensing system and data processing method of the embodiments of the present invention have the following beneficial effects:

[0037] (1) The vehicle-mounted sensing system of the present invention uses an acoustic sensing system to obtain audio data around the vehicle, and uses a traditional sensing system to obtain environmental data around the vehicle. The central processing device fuses the processing results of the acoustic sensing system and the traditional sensing system to identify obstacles around the vehicle, improving the sensing accuracy of the vehicle-mounted sensing system for obstacle information around the vehicle.

[0038] (2) The vehicle-mounted sensing system identifies obstacles in the surrounding environment of the vehicle and determines the risk coefficient of the obstacles, thereby making reasonable avoidance and adjustment based on the risk coefficient of the obstacles, improving the safety factor of the vehicle during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To better understand the above and other objects, features, advantages and functions of the present invention, reference may be made to the embodiments shown in the drawings. The same reference numerals in the drawings refer to the same components. Those skilled in the art should understand that the drawings are intended to schematically illustrate the preferred embodiments of the present invention and have no restrictive effect on the scope of the present invention. The components in the drawings are not drawn to scale.

[0040] Figure 1 It is a structural block diagram of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the principle of the first sensing device of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of the principle of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0043] Figure 4 It is a distribution schematic diagram of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0044] Figure 5 It is a schematic diagram of the principle of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0045] Figure 6 It is a schematic diagram of the principle of the acoustic frame of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0046] Figure 7 It is a schematic diagram of obstacle information determined by the first sensing device of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0047] Figure 8 It is an acoustic frame schematic diagram of a sound source obtained by the second sensing device of a vehicle-mounted sensing system according to an embodiment of the present invention;

[0048] Figure 9Schematic diagram of the central processing device of the vehicle-mounted sensing system according to an embodiment of the present invention for fusing the information of the first sensing device and the second sensing device;

[0049] Figure 10 Schematic diagram of the principle for determining the wind speed of the vehicle surrounding environment by the vehicle-mounted sensing system according to an embodiment of the present invention;

[0050] Figure 11 Schematic diagram of the principle for applying the wind speed of the vehicle-mounted sensing system according to an embodiment of the present invention;

[0051] Figure 12 Block diagram of the principle of the vehicle-mounted sensing system according to an embodiment of the present invention. Detailed implementation manners

[0052] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following.

[0053] As used herein, the term "including" and its variants mean open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "an exemplary embodiment" and "an embodiment" mean "at least one exemplary embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions hereinafter.

[0054] To at least partially solve one or more of the above problems and other potential problems, embodiments of the present disclosure propose a vehicle-mounted sensing system, including a first sensing device, a second sensing device, and a central processing device. The first sensing device is configured to obtain environmental data around the vehicle and determine obstacle information around the vehicle based on the environmental data. The obstacle information includes the size, coordinates, and motion state of the obstacle. The second sensing device is configured to obtain audio data around the vehicle and determine sound source information around the vehicle based on the audio data. The sound source information includes the acoustic coordinates and acoustic size of the sound source. The central processing device is electrically connected to the first sensing device and receives the obstacle information determined by the first sensing device. The central processing device is electrically connected to the second sensing device and receives the sound source information determined by the second sensing device. The central processing device processes the received obstacle information and sound source information and identifies obstacles around the vehicle and / or determines the wind speed of the vehicle's environment.

[0055] Specifically, as Figure 1 shown, the vehicle-mounted sensing system includes a first sensing device 100, a second sensing device 200, and a central processing device 300. The first sensing device 100 and the second sensing device 200 are respectively electrically connected to the central processing device 300, and the central processing device 300 is electrically connected to the main controller of the vehicle to realize the communication between the vehicle-mounted sensing system and the vehicle main controller. Further, by sending the obstacle information around the vehicle or the environmental wind speed of the vehicle identified by the vehicle-mounted sensing system to the vehicle main controller, the vehicle main controller adjusts the speed and direction of the vehicle based on the above results. The central processing device of the vehicle-mounted sensing system includes a receiving module and a processing module. The receiving module is electrically connected to the first sensing device 100 and the second sensing device 200. The receiving module receives the obstacle information and sound source information around the vehicle from the first sensing device 100 and the second sensing device 200, and the processing module fuses the obstacle information and the sound source information to accurately identify the obstacles around the vehicle or determine the environmental wind speed where the vehicle is located.

[0056] In this embodiment, the second sensing device includes an acoustic electronic control unit and at least one microphone array. The at least one microphone array is fixedly arranged on the vehicle and is used to acquire the audio data around the vehicle. The acoustic electronic control unit is arranged inside the vehicle. The acoustic electronic control unit is electrically connected to the at least one microphone array and is electrically connected to the central processing device. The acoustic electronic control unit is used to receive and process the audio data acquired by the at least one microphone array to determine the sound source information and send the sound source information to the central processing device.

[0057] As Figures 1 to 3As shown, the second sensing device 200 adopts an acoustic module, including an acoustic electronic control unit 230 and at least one microphone array. In this embodiment, the microphone array adopts a microphone array based on MEMS (Micro-Electro-Mechanical System). The microphone array based on MEMS includes a controller power module, a plurality of MEMS microphones, DSP (Digital Signal Processing), an A2B (Autmotive Audio Bus) transceiver, and a controller power module. The MEMS microphone is a commonly used microphone in the digital audio field. One embodiment of its signal output port interface is the I2S (Integrated interchip Sound) interface. The DSP is a commonly used DSP with common acoustic interfaces in the digital audio field. The model of one specific embodiment is TMS320C6748. Embodiments of the common acoustic interfaces include the I2S (Integrated Interchip Sound) interface, or McBSP (Multichannel BufferedSerial Port), or McASP (Multichannel Audio Serial Port) interface. The output interface of the MEMS microphone is electrically connected to the acoustic interface of the DSP; the A2B transceiver is electrically connected to the acoustic interface of the DSP; the output end of the controller power module is connected to the power input ends of the DSP, the A2B transceiver, and the MEMS microphone, and supplies power to them.

[0058] In this embodiment, the acoustic electronic control unit 230 includes a controller power supply module, a DSP, a serial port-CAN (Controller Area Network) conversion module, an A2B transceiver, and a CAN transceiver. The DSP is a commonly used DSP with common acoustic interfaces in the field of digital audio, and the model of a specific embodiment thereof is TMS320C6748; embodiments of the common acoustic interfaces include I2S interfaces, McBSP, or McASP interfaces; the serial port-CAN conversion module is a commonly used conversion module in the field of automotive electronics. The internal connection relationship of the acoustic electronic control unit 230 is as follows: the output end of the controller power supply module is connected to the power input ends of the DSP, the serial port-CAN conversion module, the A2B transceiver, and the CAN transceiver, and supplies power to them. The A2B transceiver is connected to the acoustic interface and wires of the DSP; the serial port of the DSP is connected to the serial port-CAN conversion module by wires. The serial port-CAN conversion module is connected to the CAN transceiver interface by wires, and the other end of the CAN transceiver is connected to the central processing device 300. An embodiment of the CAN transceiver is CAN-FD (Controller Area Network with Flexible Data-Rate).

[0059] Specifically, as Figure 1 shown in the embodiment, the second sensing device 200 includes a first microphone array 210, a second microphone array 220, and an acoustic electronic control unit 230. The first microphone array 210 and the second microphone array 220 are respectively disposed on the body sheet metal inside the vehicle between the right B-pillar 8 of the vehicle, the left B-pillar 9 of the vehicle, and the vehicle roof 16. In other embodiments, the second sensing device 200 may further include a third microphone array, a fourth microphone array, or multiple microphone arrays. The first microphone array 210, the second microphone array 220, and other multiple microphone arrays are respectively fixedly disposed on the body sheet metal between the right A-pillar 6 of the vehicle and the roof 16, on the body sheet metal between the left A-pillar 7 of the vehicle and the roof 16, on the body sheet metal between the right A-pillar 6 of the vehicle and the floor, on the body sheet metal between the right C-pillar 12 of the vehicle and the roof 16, on the body sheet metal between the left C-pillar 13 of the vehicle and the roof 16, on the body sheet metal at the position between the right C-pillar 12 of the vehicle and the floor, on the body sheet metal at the position between the left C-pillar 13 of the vehicle and the floor, under the front row seats, at any one position or a combination of multiple positions.

[0060] Specifically, the acoustic electronic control unit 230 is fixedly arranged on the armrest sheet metal inside the vehicle to increase the stability of the second sensing device 200 and facilitate the electrical connection between the acoustic electronic control unit 230 and the central processing device 300. The fixing method of the acoustic electronic control unit 230 is clamping, bonding, screw connection, bolt connection or welding commonly used in the vehicle field. In another embodiment, the acoustic electronic control unit 230 can also be fixedly arranged on the lower instrument panel inside the vehicle, etc., as long as the fixing of the acoustic electronic control unit 230 and the electrical connection with the central processing device 300 can be achieved.

[0061] In this embodiment, the central processing device 300 includes a controller power module, five CAN transceivers and an embedded computer or a system on chip computer system. The embedded computer or the system on chip computer system is commonly used in the field of automotive electronics and is a SOC (System on Chip) with a CAN interface. A preferred embodiment of the CAN is CAN-FD. The model of a specific embodiment of the embedded computer or the system on chip computer system is TDA4VM. The internal connection relationship of the central processing device 300 is that the output end of the controller power module is connected to the power input ends of the five CAN transceivers and the embedded computer or the system on chip computer system and supplies power to them. The five CAN transceivers are respectively wire-connected to the CAN interfaces of the embedded computer or the system on chip computer system. The central processing device 300 is fixed under the vehicle's center console or in the trunk. The preferred position is under the vehicle's center console. The fixing method is clamping, bonding, screw connection, bolt connection or welding commonly used in the vehicle field.

[0062] In this embodiment, the first sensing device 100 includes a millimeter-wave radar module, a camera module and a lidar module. Among them, the millimeter-wave radar module, the camera module and the lidar module are respectively connected to the central processing device. The receiving module of the central processing device 300 is electrically connected to the millimeter-wave radar control unit, the camera control unit and the lidar control unit. The information received by the receiving module from the control units of the respective modules of the first sensing device 100 and the information of the acoustic electronic control unit 230 of the second sensing device 200 are fused through the processing module of the central processing device 300 to identify obstacles around the vehicle.

[0063] In another embodiment, the first sensing device 100 may further include a millimeter-wave radar module, a camera module, and a first processing unit. The millimeter-wave radar module is fixedly arranged on the vehicle and is used to obtain environmental data of a preset area outside the vehicle; the camera module is fixedly arranged on the vehicle and is used to obtain image data around the vehicle; the first processing unit is electrically connected to the millimeter-wave radar module and the camera module. The first processing unit is used to receive and process the recognition results of the millimeter-wave radar module and the camera module to determine obstacle information around the vehicle, and send the obstacle information to the central processing device 300. In other embodiments, the first sensing device 100 may further include a millimeter-wave radar module, a lidar module, a camera module, and a first processing unit. The lidar module includes being fixedly arranged on the vehicle and is used to obtain information such as the position, motion state, and shape of obstacles in a preset area outside the vehicle, and send the above information to the first processing unit. The first processing unit determines the obstacle information around the vehicle by processing the recognition results of the millimeter-wave radar module, the lidar module, and the camera module.

[0064] The camera module is a commonly used camera module in the field of intelligent driving vehicles. It consists of multiple camera modules and a panoramic view controller. The output port of the camera module controller is a CAN interface and is electrically connected to the central processing device 300 through the CAN interface. In this embodiment, the camera module includes a first camera 111, a second camera 112, and a camera control unit 101. The first camera 111 is arranged at the midpoint of the radiator grille 2, and the second camera 112 is fixedly arranged at the midpoint of the rear bumper 14 at the rear end of the vehicle.

[0065] In other embodiments, the camera module may further include a first camera 111, a second camera 112, a third camera 113, a fourth camera 114, and a camera control unit 101. The first camera 111 is arranged at the midpoint of the radiator grille 2, the second camera 112 is arranged at the midpoint of the rear bumper 14 at the rear end of the vehicle, the third camera 113 is arranged on the right rearview mirror housing 4, and the fourth camera 114 is symmetrically arranged with the third camera 113 on the left rearview mirror housing 5. Among them, the fixing methods of multiple cameras are commonly used in the vehicle field, such as snap connection, bonding, screw connection, bolt connection, or welding. As Figure 5 shown, the first camera 111 obtains image data within the first camera area 1110 at the front end of the vehicle, the second camera 112 obtains image data within the second camera area 1120 at the rear end of the vehicle, the third camera 113 obtains image data within the third camera area 1130 on the right side of the vehicle, the fourth camera 114 obtains image data within the fourth camera area 1140 on the left side of the vehicle. The camera control unit 101 controls the opening and closing of the above cameras, and receives and processes the image data obtained by multiple cameras to identify obstacles around the vehicle.

[0066] In another embodiment, the camera module may further include five, six or more cameras and a camera control unit 101. The camera control unit 101 is electrically connected to the multiple cameras and controls the operations of the multiple cameras. The multiple cameras are fixedly arranged at any one or a combination of multiple positions, including the midpoint position of the radiator grille 2, the position of the right rearview mirror housing 4, the position of the left rearview mirror housing 3, and the midpoint position of the rear bumper 14, as long as the sum of the imaging areas of the cameras can cover the circumferential space of the vehicle.

[0067] In some embodiments, the millimeter-wave radar module is arranged at any one or a combination of multiple positions, including the midpoint position of the front bumper 1 of the vehicle, the two ends of the front bumper 1 of the vehicle, the midpoint position of the rear bumper 14 of the vehicle, the position of the left side body 11 of the vehicle, and the position of the right side body 10 of the vehicle.

[0068] Specifically, the millimeter-wave radar module is a commonly used millimeter-wave radar module in the field of intelligent driving vehicles, which includes multiple millimeter-wave radar modules and a millimeter-wave radar control unit. The output port of the millimeter-wave radar control unit is a CAN interface and is electrically connected to the central processing device 300 through the CAN interface.

[0069] In some embodiments, the millimeter-wave radar module includes a millimeter-wave radar control unit and at least one millimeter-wave radar. The millimeter-wave radar control unit is electrically connected to the at least one millimeter-wave radar, and the millimeter-wave radar control unit receives and processes the signals of the at least one millimeter-wave radar.

[0070] As Figure 1 shown, the millimeter-wave radar module includes a first millimeter-wave radar 121, a second millimeter-wave radar 122 and a millimeter-wave radar control unit 102. The millimeter-wave radar control unit 102 is electrically connected to the first millimeter-wave radar 121 and the second millimeter-wave radar 122, controls the opening and closing of each millimeter-wave radar, and receives and processes the environmental data of the vehicle from each millimeter-wave radar.

[0071] In other embodiments, as Figure 4 and Figure 5As shown in the figure, the millimeter-wave radar module may further include a first millimeter-wave radar 121, a second millimeter-wave radar 122, a third millimeter-wave radar 123, a fourth millimeter-wave radar 124, a fifth millimeter-wave radar 125, a sixth millimeter-wave radar 126, and a millimeter-wave radar control unit 102. Among them, the first millimeter-wave radar 121 is fixedly arranged at the midpoint position of the front bumper 1 of the vehicle, the second millimeter-wave radar 122 is fixedly arranged at the midpoint position of the rear bumper 14 of the vehicle, the third millimeter-wave radar 123 is arranged at the left fender at the front end of the vehicle, the fourth millimeter-wave radar 124 is arranged at the right fender at the front end of the vehicle, the fifth millimeter-wave radar 125 is arranged at the right side panel 10 of the vehicle, and the sixth millimeter-wave radar 126 is arranged at the left side panel 11 of the vehicle. The millimeter-wave radar control unit 102 receives and processes the environmental data of each area around the vehicle obtained by each millimeter-wave radar, including a first millimeter-wave radar area 1210, a second millimeter-wave radar area 1220, a third millimeter-wave radar area 1230, a fourth millimeter-wave radar area 1240, a fifth millimeter-wave radar area 1250, and a sixth millimeter-wave radar area 1260.

[0072] In other embodiments, the millimeter-wave radar module may further include a plurality of millimeter-wave radars and a millimeter-wave radar control unit 102. The fixed positions of the plurality of millimeter-wave radars include, but are not limited to: the front bumper 1 of the vehicle, the radiator grille 2, the front windshield 3, the outer shell of the right rearview mirror 4, the outer shell of the left rearview mirror 5, the exterior panel of the right B-pillar 8 of the vehicle, the exterior panel of the left B-pillar 9 of the vehicle, the right side panel 10 of the vehicle, the left side panel 11 of the vehicle, the body sheet metal at the right C-pillar 12 of the vehicle, the body sheet metal at the left C-pillar 13 of the vehicle, the rear bumper 14 of the vehicle, and the vehicle rear lid 15.

[0073] In this embodiment, the first sensing device further includes a lidar module. The lidar module is electrically connected to the first processing unit. The lidar module is used to identify obstacles based on the environmental data around the vehicle and send the identification to the first processing unit.

[0074] The lidar module is a commonly used lidar module in the field of intelligent driving vehicles and is composed of one or more lidars and a lidar control unit. The output port of the lidar control unit is a CAN interface. In this embodiment, the lidar module includes a first lidar 131 and a lidar control unit 103. The first lidar 131 is arranged in the interior panel of the vehicle roof 16.

[0075] In another embodiment, the first sensing device may further include a lidar module, a millimeter-wave radar module, and a camera module. Each module is respectively connected to the central processing device 300, and the central processing device 300 fuses the recognition results of each module to determine static obstacles and dynamic obstacles.

[0076] On the other hand, the present invention provides a data processing method using the vehicle-mounted sensing system as described above, including:

[0077] Step S1, obtaining environmental data around the vehicle by using a first sensing device, and determining obstacle information around the vehicle based on the environmental data, where the obstacle information includes the size, coordinates, and motion state of the obstacle.

[0078] Step S2, obtaining audio data around the vehicle by using a second sensing device, and determining sound source information around the vehicle based on the audio data, where the sound source information includes the acoustic coordinates and acoustic size of the sound source.

[0079] Step S3, electrically connecting a central processing device to the first sensing device, and receiving the obstacle information determined by the first sensing device, electrically connecting the central processing device to the second sensing device, and receiving the sound source information determined by the second sensing device, where the central processing device processes the obstacle information and the sound source information and identifies obstacles around the vehicle or determines the ambient wind speed of the vehicle.

[0080] In some embodiments, the step of determining obstacle information around the vehicle by using the first sensing device in step S1 includes:

[0081] Step S1a, obtaining environmental data by using a millimeter-wave radar module and a camera module; the first sensing device uses a common sensing algorithm in the field of intelligent driving to perform target recognition on millimeter-wave radar, cameras, and lidar.

[0082] Step S1b, using a first processing unit to fuse the recognition results of the millimeter-wave radar module and the camera module to determine the size, coordinates, and motion state of obstacles around the vehicle; multi-sensor fusion uses a common multi-sensor fusion algorithm in the field of intelligent driving to fuse the results recognized by millimeter-wave radar, cameras, and lidar.

[0083] Among them, the obstacle information includes static obstacle information output and dynamic obstacle information output. As Figure 7 shown, the following information of the dynamic obstacle calculated in the multi-sensor fusion of the dynamic obstacle output: (x TBV , y TBV ) is the center position of the perception frame of the target recognized by the traditional perception algorithm; (L TBV , W TBV ) is the length and width of the perception frame of the recognized target; (V XTBV , V YTBV ) is the speed in the x-direction and y-direction of the recognized perception frame. The static obstacle output is the following information of the static obstacle calculated in the multi-sensor fusion: (x SOj , y SOj) is the center position of the static obstacle perception frame recognized by the traditional perception algorithm. As Figure 10 shown, in a specific real-time example, there are a total of 1, 2, ……, N static obstacles, and the center positions of the corresponding static obstacle perception frames are (x SO1 , y SO1 ), (x SO2 , y SO2 ), ……, (x SON , y SON ) respectively.

[0084] In some embodiments, the step of using the second sensing device to obtain the audio data around the vehicle and determining the sound source information around the vehicle based on the audio data in step S2 includes:

[0085] Step S2a, using at least one microphone array to collect an environmental audio data set. Each MEMS-based microphone array module of the second sensing device detects the sound sources around the vehicle respectively.

[0086] Step S2b, using the acoustic electronic control unit to obtain the environmental audio data set obtained by at least one microphone array, and determining the sound source feature recognition algorithm based on the neural network algorithm.

[0087] Step S2c, using at least one microphone array to collect the audio data around the vehicle, using the acoustic electronic control unit to receive the audio data around the vehicle and identifying the sound source features based on the sound source feature recognition algorithm. The sound source features include the components, categories and conditions of the sound-emitting parts.

[0088] Step S2d, the acoustic electronic control unit determines the acoustic size of the sound source according to the sound source features, and determines the acoustic coordinates of the sound source based on the sound source localization algorithm according to the audio data around the vehicle.

[0089] The acoustic electronic control unit takes the sound source signals (i.e., audio data) detected by each MEMS-based microphone array as input, and uses the sound source feature recognition algorithm and the sound source localization algorithm to obtain the features and position information of the specific sound sources related to the vehicle. The sound source feature recognition algorithm in the vehicle feature sound recognition is obtained by training an acoustic neural network model with the vehicle environmental audio data set. The vehicle environmental audio includes the vehicle noise audio data set, that is, audio files of various vehicles making sounds under various driving conditions, various road conditions and various weather environments; the true value of the samples in the vehicle noise audio data set is the sound features of the sound source signals of the noises, specifically including the components, component categories and vehicle conditions of the sounds. The acoustic neural network model is a commonly used neural network model in the field of intelligent algorithms. A specific embodiment is a BP or RNN neural network model.

[0090] In some embodiments, the categories of the sound source features determined by the second sensing device include any one or a combination of tire noise, engine noise, exhaust noise, rearview mirror wind noise, and in-vehicle speaker audio, where the in-vehicle speaker audio includes the sirens emitted by ambulances, fire trucks, and police cars. In some embodiments, the sound source features further include the vehicle model corresponding to the sound source. In some embodiments, the conditions of the sound source features include emergency braking, rapid acceleration, and steady driving.

[0091] Specifically, taking the audio data obtained by the second sensing device from the surrounding audio of other vehicles around the vehicle as an example, the sound source information determined by the second sensing device is the recognition of the sound source features of other vehicles, and the recognition result is the specific sound source feature NoiseVehID(Part, Type, Condition).

[0092] The sound source feature NoiseVehID(Part, Type, Condition) consists of three components: the part (Part) that emits the sound, the category (Type), and the condition (Condition). The said part (Part) includes: tires, engines, exhausts, rearview mirrors, speakers / alarms, etc. The said category (Type) includes: passenger cars, commercial buses, trucks, ambulances, fire trucks, police cars, etc. The said condition (Condition) includes: emergency braking, rapid acceleration, etc.

[0093] Table 1, Values of the Parts of the Sound Source

[0094]

[0095]

[0096] Table 2, Values of the Categories of the Sound Source

[0097] Value of Type Explanation of value meaning 1 Passenger car 2 Commercial bus 3 Truck 4 Ambulance 5 Fire truck 6 Police car 7 Sound source cannot be identified

[0098] Table 3, Values of the Conditions of the Sound Source

[0099] Value of Condition Explanation of value meaning 1 Emergency braking 2 Sudden acceleration 3 Steady driving

[0100] Among them, a specific embodiment of NoiseVehID(Part, Type, Condition) is NoiseVehID(1, 3, 1), which means that the result of the recognition using the sound source feature recognition algorithm is that the sound comes from the tire noise of a truck in emergency braking. Based on the recognition result, the acoustic size of the corresponding part is further determined.

[0101] In some embodiments, the sound source information further includes the acoustic frame of the sound source, where the acoustic frame is the spatial distribution of the acoustic size of the sound source. In some embodiments, the steps for the acoustic electronic control unit to determine the acoustic frame of the sound source include: determining the acoustic coordinates of the sound source and the acoustic coordinates at the previous moment; using the direction determined by connecting the current acoustic coordinates of the sound source and the acoustic coordinates at the previous moment as the length direction of the acoustic frame, using the current acoustic coordinates of the sound source as the center of the acoustic frame, and using the length and width of the acoustic size of the sound source as the length and width of the acoustic frame. In some embodiments, when the sound source is a whole vehicle, the steps for the acoustic electronic control unit to determine the acoustic frame include: when the relative distance change between the center coordinates of the acoustic frames of multiple sound sources is less than a set first threshold, determining that the multiple sound sources belong to the same vehicle; using the smallest rectangular frame including the acoustic frames of multiple sound sources belonging to the same vehicle as the acoustic frame of the whole vehicle.

[0102] The sound source information further includes the acoustic coordinates of the sound source, that is, the sound source characteristic sound localization. Taking the vehicle characteristic sound localization as an example, the vehicle characteristic sound localization outputs the localization information of the vehicle corresponding to the sound source detected in the vehicle characteristic sound recognition. The localization information of a specific sound source includes the acoustic coordinates (x NV , y NV ) of the vehicle corresponding to the specific sound source; the acoustic coordinates (x NV , y NV ) are the coordinate values calculated by using the common sound source localization algorithms in the field of microphone array localization.

[0103] The principle and steps of vehicle acoustic target recognition are as follows:

[0104] According to the NoiseVehID(Part, Type, Condition) obtained by sound source characteristic recognition and the coordinates (x NV , y NV ) of the specific sound source obtained by sound source localization, draw the acoustic frame of the component. The specific steps are as follows:

[0105] a) As Figure 6 shown, obtain the moving direction line of the sound source according to the sound source position (x NVc , y NVc ) at the current moment Tcur and the sound source position (x NVp , y NVp ) at the previous moment Tpre.

[0106] b) Determine the sounding component according to "Part" in NoiseVehID(Part, Type, Condition), and determine the category to which the component belongs according to "type" in NoiseVehID(Part, Type, Condition). Then determine the length and width of the acoustic frame based on the component and its category. For example, when NoiseVehID(1, 3, 1), Part = 1, indicating that the detected sound comes from tire noise; type = 3, indicating that the detected sound comes from a truck, that is, the sound source is a truck tire. Determine the width W of the component acoustic frame according to the average size of commonly used truck tires in the vehicle field NVBP and the length L of the component acoustic frame NVBP .

[0107] c) With the sound source position (x NVc , y NVc ) at the current moment Tcur as the center, and with W NVBP and L NVBP as the width and length of the rectangle, draw a rectangle along the sound source movement direction line obtained in a). This rectangle is the component acoustic frame

[0108] d) Output the component acoustic frame obtained in c). When Part = 1 and Condition = 2 in NoiseVehID(Part, Type, Condition), that is, tire noise during rapid acceleration, output the tire noise frequency f accN during rapid vehicle acceleration; when Part = 1 and Condition = 1 in NoiseVehID(Part, Type, Condition), that is, tire noise during emergency braking, output the recognized tire noise frequency f braN during emergency vehicle braking

[0109] Draw the vehicle acoustic frame based on the component acoustic frame obtained in the previous steps. The specific steps are as follows

[0110] 1) Component acoustic frame clustering. Detect the change value of the relative distance between each component acoustic frame obtained in d). When the change value of the relative distance between the acoustic frames is less than the preset threshold NoiseVehDis, determine that these component acoustic frames come from the same vehicle

[0111] 2) Determine the vehicle acoustic frame. Include all the component acoustic frames determined to come from the same vehicle in 1) with a smallest rectangle. This smallest rectangle is the vehicle acoustic frame. Refer to Figure 8 as shown. The length of the vehicle acoustic frame is L NBV ; the width of the vehicle acoustic frame is W NBV ; the center of the vehicle acoustic frame is the center (x NBV , yNBV )。

[0112] Specifically, as Figure 8 shown, when the target obstacle of the vehicle is a whole vehicle, the vehicle-mounted sensing system determines the engine acoustic frame 201, the exhaust pipe acoustic frame 202, the first tire acoustic frame 203, the second tire acoustic frame 204, the first rearview mirror acoustic frame 205, the second rearview mirror acoustic frame 206, the third tire acoustic frame 207, and the fourth tire acoustic frame 208 through the second sensing device. By setting a first threshold value, the change in the relative distance between each acoustic frame is judged. In this embodiment, the distance change between each acoustic frame is determined by the distance between the central coordinates of each component acoustic frame. When the distance change between the central coordinates of each acoustic frame is less than the set first threshold value, it is judged that the components corresponding to each acoustic frame belong to the same vehicle, and the acoustic frame of the whole vehicle is determined according to the position and spatial distribution of each acoustic frame, so that the length and width of the acoustic frame of the whole vehicle are the minimum values including each acoustic frame.

[0113] In some embodiments, when the central processing device judges that the distance between the acoustic coordinate of the sound source and the central coordinate of the obstacle is less than the set second threshold value, the obstacle and the object corresponding to the sound source are the same obstacle, and the spatial distribution of the geometric size of the obstacle is determined according to the minimum rectangular frame including the acoustic frame and the obstacle sensing frame of the obstacle.

[0114] Specifically, the central processing device includes a receiving module and a processing module. The receiving module receives the obstacle information sent by the first sensing device and the acoustic information sent by the second sensing device. The processing module fuses the above information obtained by the receiving module to identify the obstacles around the vehicle and determine the environmental wind speed of the vehicle. Among them, identifying the obstacles around the vehicle includes correcting the obstacle information of the first sensing device through the acoustic information sent by the second sensing device. As Figure 9 shown, taking a target obstacle around the vehicle as an example, the first sensing device senses the target obstacle and outputs obstacle information for the target obstacle, including coordinates (x TBV , y TBV ), and the length L TBV of the sensing frame size data and the width W TBV , and the speeds (v xTBV , v yTBV ) of the sensing frame in the X direction and the Y direction, where the sensing frame is the spatial distribution of the obstacle size determined by the first sensing device; the second sensing device senses the target obstacle and outputs the acoustic coordinates and acoustic size of the whole vehicle for the target obstacle. Among them, the acoustic coordinates are (x NBV , y NBV ), and the acoustic size data includes the length L NBV and the width W NBV. The processing module of the central processing unit compares the coordinates (x TBV , y TBV ) with the acoustic coordinates (x NBV , y NBV ). When the distance between the two points is less than the set second threshold, it is determined that the target obstacle measured by the first sensing device and the target obstacle measured by the second sensing device are the same target. Then, based on the size distribution of the sensing frame determined by the first sensing device and the size distribution of the acoustic frame determined by the second sensing device, the spatial distribution of the size of the target obstacle is re-identified, and a smallest rectangle is used to enclose both the acoustic frame determined by the second sensing device and the sensing frame determined by the first sensing device. This smallest rectangle is the finally fused sensing frame. The length L BV and width W BV of this rectangle are the length and width of the finally fused vehicle perception frame; the center coordinates (x BV , y BV ) of this rectangle are the center coordinates of the finally fused vehicle perception frame. The speed of the finally fused vehicle perception frame is (v xBV , v yBV ), where v xBV is equal to v xTBV , and v yBV is equal to v yTBV .

[0115] In some embodiments, the sound source information further includes the center coordinates of the wind speed sound source, the wind speed sound source frequency, and the wind speed sound source intensity.

[0116] Taking the sound source signals (i.e., audio data) detected by multiple microphone arrays of the acoustic electronic control unit of the second sensing device of the vehicle-mounted sensing system as input, a wind speed sound source feature recognition algorithm and a sound source localization algorithm are used to obtain the feature and position information of a specific sound source. Specifically, it is divided into the following steps:

[0117] Wind speed sound source feature recognition; the recognition result obtained by wind speed sound source feature recognition is whether the sound source can be used as a wind speed sound source. The wind speed sound source feature recognition algorithm is obtained by training an acoustic neural network model with a wind noise audio set. The wind noise audio set is an audio file of the air vortex sound when the wind blows through different static obstacles at different wind speeds. The ground truth of the samples in the wind noise audio set is: a) can be used as a wind speed sound source; b) cannot be used as a wind speed sound source; among them, the air vortex sounds corresponding to when the wind blows through columnar static obstacles such as utility poles, traffic sign poles, and tree trunks have the ground truth of a) can be used as a wind speed sound source; the ground truth of the remaining sounds is b) cannot be used as a wind speed sound source. Among them, the acoustic neural network model is a commonly used neural network model in the field of intelligent algorithms, and a specific embodiment is a BP or RNN neural network model.

[0118] Wind speed sound source localization; the output of wind speed sound source localization is the localization information of the wind speed sound source detected in the wind speed sound source feature recognition. The localization information of the wind speed sound source is the position information (x NW , y NW ); the position information (x NW , y NW ) of the specific sound source is the coordinate value calculated by the common sound source localization algorithm in the field of microphone array localization.

[0119] Wind speed sound information output; the information output in the wind speed sound information output includes: the sound frequency f of the wind speed sound source detected in the wind speed sound source feature recognition NW ; the sound intensity dB of the wind speed sound source detected in the wind speed sound source feature recognition NW ; the center coordinates (x NW , y NW ) of the wind speed sound source calculated in the wind speed sound source localization.

[0120] In some embodiments, the steps for the central processing device to determine the environmental wind speed include:

[0121] Step t1, use the first sensing device to determine the center coordinates of the wind speed sound source, and use the second sensing device to determine the coordinates of the static obstacle, taking the center of the wind speed sound source and the static obstacle as a wind speed measurement pair.

[0122] Step t2, determine the distance between the wind speed measurement pairs according to the center coordinates of the wind speed sound source and the coordinates of the static obstacle: Determine the angle between the line connecting the wind speed measurement pairs and the vehicle driving direction: In the formula, (x NWM , y NWM ) are the coordinates of the center of the wind speed sound source in the geodetic coordinate system, and (x SON , y SON ) are the coordinates of the static obstacle in the geodetic coordinate system.

[0123] Step t3, calculate the average value of the angles between multiple groups of wind speed measurement pairs and the vehicle driving direction Set the angle threshold Δθ WANM , distance threshold MaxD NM , and take the wind speed measurement pairs that satisfy D NM <MaxD NM and as the counting measurement points.

[0124] Step t4, calculate the wind noise frequency of the counting measurement points: f NWi is the frequency at which the audio emitted by the center of the wind speed sound source corresponding to the i-th counting measurement point is received, u is the speed of sound, v is the vehicle speed, αi is the included angle between the line connecting the center of the wind speed sound source at the i-th counting measurement point and the vehicle and the vehicle moving direction.

[0125] Step t5, according to the determined wind noise frequencies of multiple counting measurement points, determine the ambient wind speed of the vehicle as: Wherein, where the value range of i corresponds to the total number n of counting measurement points, dB NWi is the sound intensity corresponding to the i-th counting measurement point, v WSi is determined according to the sound source model f NWO the wind speed at the i-th counting measurement point corresponding to.

[0126] Specifically, as Figure 12 shown, the wind speed measurement point identification is divided into the following steps: perform wind speed sound source positioning through the second sensing device to obtain the position (x NWm , y NWm ) of the wind speed sound source. In this embodiment scenario, the positions of 1, 2... M wind speed sound sources are obtained, and the center coordinates of each wind speed sound source are respectively (x NW01 , y NW01 ), (x NW02 , y NW02 )... (x NWM , y NWM ). Obtain the positions of static obstacles through the first sensing device. In this embodiment scenario, the positions of N static obstacles are obtained, and the static obstacle coordinates are respectively (x SO01 , y SO01 ), (x SO02 , y SO02 )... (x SON , y SON ). According to the obtained positions of the wind speed sound source and the static obstacles, calculate and find the static obstacle closest to the position of each wind speed sound source. The wind speed sound source and its corresponding closest static obstacle are a wind speed measurement pair.

[0127] Calculate the distance D between the static obstacle and the center of the wind speed sound source in each pair of potential wind speed measurement pairs NM and the angle θ between their connection line and the vehicle driving direction WANM . If a certain static obstacle N and a certain wind speed sound source M form a pair of wind speed measurement pairs, then this wind speed measurement pair is called the wind speed measurement pair NM, and the distance D NM and the angle θ WANM of this wind speed measurement pair NM are expressed as:

[0128] D NM represents the distance between the center of the static obstacle N and the center of the sound source M;

[0129] θ WANM It represents the included angle between the line connecting the centers of the static obstacle N and the sound source M and the x-axis of the earth coordinate system.

[0130] Based on the distance D of each pair of wind speed measurement pairs NM and the angle θ WANM , identify the final counting measurement points, and the specific method is as follows:

[0131] a) Calculate the average value of the angles θ of all wind speed measurement points WANM

[0132] b) Preset the threshold values MaxD NM and Δθ WANM , where MaxD NM represents the maximum distance between the sound source and the obstacle among the available wind speed measurement pairs; Δθ WANM is the maximum angle deviation of the available wind speed measurement points.

[0133] c) When and D NM < MaxD NM , it is determined that this wind speed measurement pair is determined as a counting measurement point.

[0134] In some embodiments, the ambient wind speed direction of the vehicle is: Among them, in the formula, D Wi is the distance between the wind speed sound source center corresponding to the i-th counting measurement point and the static obstacle, n is the total number of counting measurement points, and θ WAi is the included angle between the line connecting the wind speed sound source center and the static obstacle in the i-th counting measurement point and the ground.

[0135] Specifically, the detection of the ambient wind speed of the vehicle is divided into the following steps:

[0136] Calculation of the ambient wind speed magnitude. a) According to the Doppler effect principle, the frequency calculation formula of the wind noise at the static obstacle is: f NWOi =(f NWi ×u) / (u + vcosαi), f NWi is the frequency received by the second acquisition device for the i-th wind speed measurement point identified; f NWOi is the frequency emitted at the sound source of the i-th wind speed measurement point; u is the speed of sound; v is the vehicle speed; αi is the included angle between the line connecting the sound source and the vehicle in the i-th counting measurement pair and the vehicle moving direction. b) The neural network estimates the wind speed magnitude based on the frequency and the sound source intensity, and uses the wind speed identification algorithm to identify the sound source wind speed v of each wind speed measurement point WS . Among them, the wind speed of the i-th counting measurement point is expressed as v​WSi ; The wind speed recognition algorithm is obtained by training a wind speed recognition neural network model with a wind speed and wind noise dataset. The wind speed and wind noise dataset includes the sound source intensity dB NW and the sound source frequency f NWi at different wind speeds. The true value of the sample in the wind speed and wind noise dataset is the wind speed v WS corresponding to the data. The wind speed recognition neural network model is a commonly used neural network model in the field of neural network algorithms. A specific embodiment is a BP or RNN neural network model. The recognition result obtained by the wind speed recognition algorithm is the wind speed v WSi at the i-th counting measurement point. c) Ambient wind speed estimation. The estimation formula for the ambient wind speed v WS is as follows: N is the number of counting measurement points; K Si is the weighted coefficient for ambient wind speed estimation; dB NWi is the sound intensity of the sound source at the i-th counting measurement point, and this sound intensity is obtained from the output of the wind speed sound information of the second sensing device.

[0137] Calculation of ambient wind speed direction. The calculation formula for the ambient wind speed direction θ WA is as follows: where, n is the number of counting measurement points; θ WAi is the angle of the i-th counting measurement point. If the i-th counting measurement point consists of a static obstacle N and a certain sound source M, then θ WAi = θ WANM ; K Ai is the weighted coefficient for ambient wind speed direction calculation; D Wi is the distance of the i-th counting measurement point. If the i-th wind speed measurement point consists of a static obstacle N and a certain sound source M, then D Wi = D NM ;

[0138] Output of ambient wind speed. The output of the ambient wind speed is θ WA and D NM .

[0139] In some embodiments, as Figure 11 shown, the component of the ambient wind speed in the vehicle driving direction is: v WSo = v WS × cos(θ WA - θ Ego ) - v Ego , and the component in the vertical direction is: v WSa = v WS × sin(θ WA - θ Ego ), where θEgo is the yaw angle of the vehicle, and v Ego is the speed of the vehicle. In another embodiment, v WSo is used for the vehicle control module to compensate for longitudinal control. v WSa is used for the vehicle control module to compensate for lateral control. That is, the central processing unit sends the above ambient wind speed to the vehicle control module, and the vehicle control module makes an adaptive adjustment to the vehicle driving mode based on the above results.

[0140] In some embodiments, the central processing unit processes obstacle information and sound source information to determine the danger coefficient of the obstacle: K Nt = K NC ×K NT , where K NC is the acoustic working condition danger coefficient, and K NT is the obstacle type danger coefficient. In some embodiments, the acoustic working condition danger coefficient includes an emergency acceleration coefficient, and the emergency acceleration coefficient is determined by the following formula: K acc = K accN ×f accN , K accN is the set emergency acceleration proportionality coefficient, and f accN is the emergency acceleration tire noise frequency determined by the second sensing device. In some embodiments, the acoustic working condition danger coefficient includes an emergency braking coefficient, and the emergency braking coefficient is determined by the following formula: K bra = K braN ×f braN , K braN is the set emergency braking proportionality coefficient, and f braN is the emergency braking tire noise frequency determined by the second sensing device. In some embodiments, the acoustic working condition danger coefficient includes a steady driving coefficient, and the steady driving coefficient is equal to the set steady driving proportionality coefficient.

[0141] Specifically, the calculation method of the acoustic target danger level is as follows: K Nt = K NC ×K NT , where the K NC is the acoustic working condition danger coefficient determined according to the vehicle driving condition. According to whether the vehicle is in emergency braking, emergency acceleration, and steady driving, it can be respectively valued as K acc , K bra and K sta .

[0142] K acc = K accN ×f accN , K accN is the preset emergency acceleration proportionality coefficient, and f accN is the frequency of the tire noise when the vehicle is in emergency acceleration output from the component acoustic frame.

[0143] K bra = K braN × f braN ,K braN is a preset emergency braking proportionality coefficient, and f braN is the frequency of the tire noise during vehicle emergency braking output in the component acoustic frame.

[0144] K sta = 1, K sta is a preset stable driving proportionality coefficient.

[0145] Table 4. K NC Parameter value examples

[0146] Value of Condition Explanation of value meaning of Condition <![CDATA[K NC value]]> 1 Emergency braking <![CDATA[K acc > 2 Sudden acceleration <![CDATA[K bra > 3 Steady driving <![CDATA[K sta >

[0147] The aforementioned K NT is the value of the acoustic vehicle type risk coefficient determined according to the vehicle type, which is the preset fixed threshold values K3, K4, K5, K6, K7, K8; the values of K3, K4, K5, K6, K7, K8 are determined according to the risk level of the corresponding vehicle. In one specific embodiment, the relative magnitude relationship of K3, K4, K5, K6, K7, K8 is: 1 = K3 < K4 < K5 < K6 < K7 < K8.

[0148] Table 5. K NT Parameter value examples

[0149]

[0150]

[0151] As Figure 12 shown, the central processing unit outputs the final perception result based on the information fusion determined by the first perception device and the second perception device, including the position of the perceived target object, the speed of the perceived target object, the size of the perceived target object, and the acoustic threat of the perceived target, and this acoustic threat can be used in the decision-making and planning algorithms to reasonably avoid the target object.

[0152] The various embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical applications, or the improvements to the technology in the market, or to enable other ordinary technical personnel in the technical field to understand this text.

Claims

1. A vehicle-mounted sensing system, characterized in that, Including: A first sensing device configured to acquire environmental data around the vehicle and determine obstacle information around the vehicle based on the environmental data, where the obstacle information includes the size, coordinates, and motion state of the obstacle; A second sensing device configured to acquire audio data around the vehicle and determine sound source information around the vehicle based on the audio data, where the sound source information includes the acoustic coordinates and acoustic size of the sound source; A central processing unit, which is electrically connected to the first sensing device and receives the obstacle information determined by the first sensing device, and is electrically connected to the second sensing device and receives the sound source information determined by the second sensing device. The central processing unit processes the received obstacle information and sound source information to identify obstacles around the vehicle and / or determine the ambient wind speed of the vehicle. The central processing unit is configured to determine the center coordinates of the wind speed sound source using the second sensing device, determine the coordinates of the static obstacle using the first sensing device, take the center of the wind speed sound source and the static obstacle as a wind speed measurement pair, and determine the distance between the wind speed measurement pairs according to the center coordinates of the wind speed sound source and the coordinates of the static obstacle: Determine the angle between the line connecting the wind speed measurement pairs and the vehicle's driving direction: Where x NWM and y NWM are the coordinates of the center of the wind speed sound source in the geodetic coordinate system, and x SON and y SON are the coordinates of the static obstacle in the geodetic coordinate system; calculate the average value of the angles between multiple groups of wind speed measurement pairs and the vehicle's driving direction Set the angle threshold Δθ WANM and the distance threshold MaxD NM . Consider the wind speed measurement pairs that satisfy D NM <MaxD NM and as the counting measurement points; calculate the wind noise frequency of the counting measurement points: f NWi is the frequency at which the audio emitted by the center of the wind speed sound source corresponding to the i-th counting measurement point is received, u is the speed of sound, v is the vehicle speed, and α i is the angle between the line connecting the center of the wind speed sound source of the i-th counting measurement point and the vehicle and the vehicle's moving direction; determine the ambient wind speed of the vehicle based on the determined wind noise frequencies of multiple counting measurement points as: Where where the value range of i corresponds to the total number n of counting measurement points, and dB NWi is the sound intensity corresponding to the i-th counting measurement point, and v WSi is the wind speed of the i-th counting measurement point.

2. The vehicle-mounted sensing system according to claim 1, wherein The second sensing device includes: At least one microphone array fixedly arranged on the vehicle and configured to acquire audio data around the vehicle; An acoustic electronic control unit arranged inside the vehicle, electrically connected to the at least one microphone array and electrically connected to the central processing device, where the acoustic electronic control unit is configured to receive and process the audio data acquired by the at least one microphone array to determine the sound source information and send the sound source information to the central processing device.

3. The vehicle-mounted sensing system according to claim 2, wherein The at least one microphone array is arranged at any one position or a combination of multiple positions on the body sheet metal between the B-pillar and the roof inside the vehicle, on the body sheet metal between the B-pillar and the floor inside the vehicle, on the body sheet metal between the A-pillar and the roof inside the vehicle, on the body sheet metal between the A-pillar and the floor inside the vehicle, on the body sheet metal between the C-pillar and the roof inside the vehicle, on the body sheet metal at the position between the C-pillar and the floor inside the vehicle, or under the front row seats.

4. The vehicle-mounted sensing system according to claim 2, wherein The acoustic electronic control unit is fixedly arranged on the armrest sheet metal inside the vehicle.

5. The vehicle-mounted sensing system according to claim 1, wherein The first sensing device includes: A millimeter-wave radar module fixedly arranged on the vehicle and configured to acquire environmental data around the vehicle; A camera module fixedly arranged on the vehicle and configured to acquire image data around the vehicle; A first processing unit electrically connected to the millimeter-wave radar module and the camera module, where the first processing unit is configured to receive and process the environmental data and the image data to determine obstacle information around the vehicle and send the obstacle information to the central processing device.

6. The vehicle-mounted sensing system according to claim 5, wherein, The millimeter-wave radar module is arranged at any one or a combination of multiple positions including the midpoint position of the front bumper of the vehicle, the two ends of the front bumper of the vehicle, the midpoint position of the rear bumper of the vehicle, the left side body position of the vehicle, and the right side body position of the vehicle.

7. The vehicle-mounted sensing system according to claim 6, characterized in that, The millimeter-wave radar module includes a millimeter-wave radar control unit and at least one millimeter-wave radar, where the millimeter-wave radar control unit is electrically connected to the at least one millimeter-wave radar, and the millimeter-wave radar control unit receives and processes the signals of the millimeter-wave radar.

8. The vehicle-mounted sensing system according to claim 5, characterized in that, The camera module is arranged at any one or a combination of multiple positions including the midpoint position of the radiator grille, the position of the right rearview mirror housing, the position of the left rearview mirror housing, and the midpoint position of the rear bumper.

9. The vehicle-mounted sensing system according to claim 8, characterized in that, The camera module includes a camera control unit and at least one camera. The camera control unit is electrically connected to the at least one camera, and the camera control unit receives and processes the signals of the camera.

10. The vehicle-mounted sensing system according to claim 5, characterized in that, The first sensing device further includes a lidar module. The lidar module is electrically connected to the first processing unit, and the lidar module is used to obtain the environmental data around the vehicle and send it to the first processing unit.

11. A data processing method for an in-vehicle sensing system according to any one of claims 1-10, characterized in that, Including: Obtaining the environmental data around the vehicle by using the first sensing device, and determining the obstacle information around the vehicle based on the environmental data. The obstacle information includes the size, coordinates, and motion state of the obstacle. Obtaining the audio data around the vehicle by using the second sensing device, and determining the sound source information around the vehicle based on the audio data. The sound source information includes the acoustic coordinates and acoustic size of the sound source. Electrically connect the central processing unit to the first sensing device, receive the obstacle information determined by the first sensing device, electrically connect the central processing unit to the second sensing device, and receive the sound source information determined by the second sensing device. The central processing unit processes the obstacle information and the sound source information to identify obstacles around the vehicle and / or determine the ambient wind speed of the vehicle. The central processing unit uses the second sensing device to determine the center coordinates of the wind speed sound source, and uses the first sensing device to determine the coordinates of static obstacles. Taking the center of the wind speed sound source and the static obstacle as a wind speed measurement pair, determine the distance between the wind speed measurement pairs according to the center coordinates of the wind speed sound source and the coordinates of the static obstacle: Determine the angle between the line connecting the wind speed measurement pairs and the vehicle driving direction: In the formula, x NWM and y NWM are the coordinates of the center of the wind speed sound source in the geodetic coordinate system, x SON and y SON are the coordinates of the static obstacle in the geodetic coordinate system; calculate the average value of the angles between multiple groups of wind speed measurement pairs and the vehicle driving direction Set the angle threshold Δθ WANM and the distance threshold MaxD NM . Take the wind speed measurement pairs that satisfy D NM <MaxD NM and as the counting measurement points; calculate the wind noise frequency of the counting measurement points: f NWi is the frequency at which the audio emitted by the center of the wind speed sound source corresponding to the i-th counting measurement point is received, u is the speed of sound, v is the vehicle speed, and α i is the angle between the line connecting the center of the wind speed sound source of the i-th counting measurement point and the vehicle and the vehicle's moving direction; according to the determined wind noise frequencies of multiple counting measurement points, determine the ambient wind speed of the vehicle as: Among them, Among them, the value range of i corresponds to the total number n of counting measurement points, dB NWi is the sound intensity corresponding to the i-th counting measurement point, and v WSi is the wind speed of the i-th counting measurement point.

12. The method according to claim 11, wherein The steps of obtaining the audio data around the vehicle by using the second sensing device and determining the sound source information around the vehicle based on the audio data include: Collecting an environmental audio data set by using at least one microphone array. The environmental audio data set includes audio files of various vehicles making sounds under various driving conditions, various road conditions, and various weather environments. Determining a sound source feature recognition algorithm according to the environmental audio data set by using an acoustic electronic control unit. Collecting the audio data around the vehicle by using the at least one microphone array, and using the acoustic electronic control unit to receive the audio data around the vehicle and recognize the sound source features based on the sound source feature recognition algorithm. The sound source features include the components, categories, and conditions of the sound-emitting parts. The acoustic electronic control unit determines the acoustic size of the sound source according to the sound source features, and determines the acoustic coordinates of the sound source based on the sound source localization algorithm according to the audio data around the vehicle.

13. The method according to claim 12, wherein The components of the sound source features include any one or a combination of tire noise, engine noise, exhaust noise, rearview mirror wind noise, and in-vehicle speaker audio. The in-vehicle speaker audio includes the alarms emitted by ambulances, fire trucks, and police cars.

14. The method according to claim 13, characterized in that, The categories of the sound source features include passenger cars, commercial buses, trucks, ambulances, fire trucks, police cars, and the sound source cannot be recognized.

15. The method according to claim 14, wherein The conditions of the sound source features include emergency braking, rapid acceleration, and stable driving.

16. The method according to claim 15, wherein The sound source information further includes the acoustic frame of the sound source. Among them, the acoustic frame is the spatial distribution of the acoustic size of the sound source.

17. The method according to claim 16, wherein The steps for the acoustic electronic control unit to determine the acoustic frame of the sound source include: Determining the current acoustic coordinates of the sound source and the acoustic coordinates of the previous moment. Taking the direction determined by connecting the current acoustic coordinates of the sound source and the acoustic coordinates of the previous moment as the length direction of the acoustic frame, taking the current acoustic coordinates of the sound source as the center of the acoustic frame, and taking the length and width of the acoustic size of the sound source as the length and width of the acoustic frame.

18. The method according to claim 17, wherein When the sound source is the whole vehicle, the steps for the acoustic electronic control unit to determine the acoustic frame include: When the relative distance change between the center coordinates of the acoustic frames of multiple sound sources is less than a set first threshold, it is determined that the multiple sound sources belong to the same vehicle. Taking the smallest rectangular frame including the acoustic frames of multiple sound sources belonging to the same vehicle as the acoustic frame of the whole vehicle.

19. The method according to claim 18, wherein The sound source information further includes the center coordinates of the wind speed sound source, the frequency of the wind speed sound source, and the intensity of the wind speed sound source.

20. The method according to claim 19, wherein The steps for determining the sound source information of the wind speed sound source include: Collecting a wind noise sound dataset using at least one microphone array, where the wind noise sound dataset includes audio files of air vortex sounds when the wind blows over different static obstacles at different wind speeds; Determining a wind speed feature recognition algorithm using an acoustic electronic control unit based on the wind noise sound dataset; Collecting audio data around the vehicle using the at least one microphone array, and using the acoustic electronic control unit to receive the audio data around the vehicle and determine whether it is a wind speed sound source based on the wind speed feature recognition algorithm; When it is determined that the audio data around the vehicle collected by the at least one microphone array is a wind speed sound source, the acoustic electronic control unit determines the sound source information of the wind speed sound source based on the audio data around the vehicle using a sound source localization algorithm.

21. The method according to claim 20, wherein The steps for determining the obstacle information around the vehicle using a first sensing device include: Obtaining environmental data using a millimeter-wave radar module and a camera module, and identifying static obstacles and dynamic obstacles based on the environmental data; Using a first processing unit to fuse the recognition results of the millimeter-wave radar module and the camera module to determine the size, coordinates, and motion state of the obstacles around the vehicle.

22. The method according to claim 20, characterized in that, The steps for obtaining the obstacle information around the vehicle using a first sensing device include: Obtaining environmental data using a millimeter-wave radar module, a camera module, and a lidar module, and identifying static obstacles and dynamic obstacles based on the environmental data; Using a first processing unit to fuse the recognition results of the millimeter-wave radar module, the camera module, and the lidar module to determine the size, coordinates, and motion state of the obstacles around the vehicle.

23. The method according to claim 21 or 22, characterized in that, When the central processing device determines that the acoustic coordinates of the sound source are less than a set second threshold from the center coordinates of the obstacle, the obstacle and the sound source are the same obstacle, and the spatial distribution of the geometric size of the obstacle is determined based on the minimum rectangular frame including the acoustic frame of the sound source and the obstacle perception frame.

24. The method according to claim 11, wherein The environmental wind speed direction of the vehicle is: Among them, In the formula, D Wi is the distance between the wind speed sound source center corresponding to the i-th counting measurement point and the static obstacle, n is the total number of counting measurement points, and θ WAi is the angle between the line connecting the wind speed sound source center and the static obstacle in the i-th counting measurement point and the ground.

25. The method according to claim 24, wherein The component of the ambient wind speed in the vehicle's driving direction is: v WSo = v WS × cos(θ WA - θ Ego ) - v Ego , and the component in the vertical direction is: v WSa = v WS × sin(θ WA - θ Ego ), where θ Ego is the yaw angle of the vehicle, and v Ego is the speed of the vehicle.

26. The method according to claim 25, wherein The central processing unit processes the obstacle information and the sound source information to determine the danger coefficient of the obstacle: K Nt = K NC × K NT , where K NC is the danger coefficient of the acoustic working condition, and K NT is the danger coefficient of the obstacle type.

27. The method according to claim 26, wherein The acoustic condition risk coefficient includes a sharp acceleration coefficient, and the sharp acceleration coefficient is determined by the following formula: K acc = K accN × f accN , where K accN is the set sharp acceleration proportionality coefficient, and f accN is the sharp acceleration tire noise frequency determined by the second sensing device.

28. The method according to claim 27, wherein The acoustic working condition risk coefficient includes an emergency braking coefficient, and the emergency braking coefficient is determined by the following formula: K bra = K braN × f braN , where K braN is the set emergency braking proportionality coefficient, and f braN is the emergency braking tire noise frequency determined by the second sensing device.

29. The method according to claim 28, wherein The acoustic working condition danger coefficient includes a stable driving coefficient, and the stable driving coefficient is equal to a set stable driving proportionality coefficient.

Images