Acoustic detection method and device in a driverless driving environment

Abstract

The embodiment of the application provides an acoustic detection method and device in an unmanned driving environment, the acoustic detection method comprising: acquiring an acoustic signal of an environment where an unmanned vehicle is located at a current time; performing multi-scale space decomposition on the acoustic signal to obtain a plurality of groups of decomposition coefficients of the acoustic signal in different scale spaces; performing acoustic feature extraction on each group of decomposition coefficients respectively to obtain a plurality of feature vectors which are the same in number as the groups of decomposition coefficients; inputting the plurality of feature vectors into a plurality of trained neural network models respectively to output classification results of identifying a sound-emitting object in the environment in different scale spaces; and determining whether the sound-emitting object is a preset type of vehicle according to the classification results in all scale spaces. The method can realize identification of specific sound-emitting vehicles such as ambulances and police cars by the unmanned vehicle using acoustic signals, so as to perform corresponding control operations such as avoidance.

Description

Technical Field

[0001] This application relates to the field of unmanned driving control technology, and in particular to an acoustic detection method and device in an unmanned driving environment. Background Technology

[0002] In autonomous driving environments, the conditions are often complex. Currently, autonomous vehicles typically perceive their surroundings using millimeter-wave radar, lidar, and cameras—all based on visual perception systems. According to the Road Traffic Safety Law, vehicles must yield to emergency vehicles such as police cars and ambulances. However, relying solely on visual perception signals is insufficient for autonomous vehicles to effectively perceive information about these emergency vehicles. Acoustic signals, as another form of perception, can also provide rich environmental information and, under specific conditions, offer more accurate information about the external environment for autonomous vehicles. Summary of the Invention

[0003] In view of this, embodiments of this application provide an acoustic detection method and apparatus for unmanned driving environments.

[0004] In a first aspect, embodiments of this application provide an acoustic detection method for an unmanned driving environment, including:

[0005] Acquire the acoustic signals of the environment in which the autonomous vehicle is located at the current moment;

[0006] The acoustic signal is decomposed into multiple scales to obtain multiple sets of decomposition coefficients at different scales.

[0007] Acoustic features are extracted from each group of decomposition coefficients to obtain multiple feature vectors with the same number of groups of decomposition coefficients.

[0008] The multiple feature vectors are respectively input into multiple trained neural network models, and the classification results of sound-emitting objects in the environment are output at different scales.

[0009] Based on the classification results across all scales, determine whether the sound-emitting object is a preset type of vehicle.

[0010] In some embodiments, the multi-scale spatial decomposition of the acoustic signal to obtain multiple sets of decomposition coefficients of the acoustic signal at different scales includes:

[0011] The acoustic signal is subjected to binary wavelet decomposition based on frequency to obtain multiple sets of wavelet coefficients and a set of residual coefficients in different scale spaces; the multiple sets of wavelet coefficients and the set of residual coefficients together constitute multiple sets of decomposition coefficients of the acoustic signal.

[0012] In some embodiments, each of the neural network models is constructed using a conditionally restricted Boltzmann machine, and the training process of the plurality of neural network models includes:

[0013] P+1 feature vectors with classification labels are used as the first training samples and are respectively input into P+1 first conditionally restricted Boltzmann machines. The P+1 first conditionally restricted Boltzmann machines are trained unsupervised using a preset learning algorithm to obtain the network parameters of each first conditionally restricted Boltzmann machine; where P is an integer, and the P+1 feature vectors are obtained by feature extraction from P sets of wavelet coefficients and 1 set of residual coefficients;

[0014] The label vectors corresponding to the P+1 feature vectors are used as the second training samples and are respectively input into the P+1 second conditionally restricted Boltzmann machines. The P+1 second conditionally restricted Boltzmann machines are unsupervised trained using a preset learning algorithm to obtain the network parameters of each second conditionally restricted Boltzmann machine.

[0015] Obtain the high-order feature information of each feature vector in the P+1 feature vectors, and the high-order feature information of each label vector in the P+1 label vectors;

[0016] The higher-order feature information of the feature vector and the label vector are used as the input and output of the recurrent neural network, respectively, and supervised training is performed to obtain the network parameters of the recurrent neural network.

[0017] The first conditionally restricted Boltzmann machine, the recurrent neural network, and the second conditionally restricted Boltzmann machine, which have obtained network parameters, are stacked from bottom to top to build a deep restricted Boltzmann machine as a trained neural network model.

[0018] In some embodiments, determining whether the sound-emitting object is a preset type of vehicle based on the classification results across all scale spaces includes:

[0019] The classification results across all scales are weighted and averaged to obtain a combined classification result.

[0020] The vehicle type corresponding to the maximum value is selected from the combined classification results to determine whether the sound-emitting object is a preset type of vehicle.

[0021] In some embodiments, the preset type of vehicle is any one of a police car, a fire truck, and an ambulance.

[0022] Secondly, embodiments of this application also provide a method for controlling an unmanned vehicle, including:

[0023] The above-mentioned acoustic detection method in the autonomous driving environment is used to determine whether the sound-emitting object in the environment where the autonomous vehicle is located is a preset type of vehicle.

[0024] When the sound-emitting object is determined to be a vehicle of a preset type, the distance from the preset type vehicle to the driverless vehicle is calculated based on the acoustic signals acquired in real time.

[0025] When a vehicle of the preset type is detected to be moving within a preset distance range directly behind the vehicle, an avoidance operation is performed.

[0026] In some embodiments, when it is determined that the sound-emitting object is a preset type of vehicle, the method further includes:

[0027] Continuously acquire multiple acoustic signals from the environment at subsequent consecutive moments;

[0028] When multiple subsequent acoustic signals are detected to contain a specific frequency signal emitted by the preset type of vehicle, the amplitude change of the specific frequency signal is used to determine whether the preset type of vehicle is approaching the driverless vehicle.

[0029] If approaching the driverless vehicle, the distance between the preset type of vehicle and the driverless vehicle is calculated based on the acoustic signals acquired in real time.

[0030] Thirdly, embodiments of this application also provide an acoustic detection device for an unmanned driving environment, comprising:

[0031] The acquisition module is used to acquire the acoustic signals of the environment in which the autonomous vehicle is located at the current moment;

[0032] The decomposition module is used to perform multi-scale spatial decomposition on the acoustic signal to obtain multiple sets of decomposition coefficients of the acoustic signal at different scales.

[0033] The feature extraction module is used to extract acoustic features from each group of decomposition coefficients to obtain multiple feature vectors with the same number of decomposition coefficient groups.

[0034] The model recognition module is used to input the multiple feature vectors into multiple trained neural network models respectively, and output the classification results of the sound-emitting objects in the environment at different scales.

[0035] The determination module is used to determine whether the sound-emitting object is a preset type of vehicle based on the classification results in all scale spaces.

[0036] Fourthly, embodiments of this application also provide an unmanned vehicle, which includes a sound sensing unit, a processor, and a memory. The sound sensing unit is used to acquire environmental acoustic signals, the memory stores a computer program, and the processor is used to execute the computer program to implement the above-described acoustic detection method or unmanned vehicle control method in an unmanned driving environment.

[0037] Fifthly, embodiments of this application also provide a readable storage medium storing a computer program, which, when executed on a processor, implements the aforementioned acoustic detection method or autonomous vehicle control method in an autonomous driving environment.

[0038] The embodiments of this application have the following beneficial effects:

[0039] The acoustic detection method in the autonomous driving environment of this application embodiment acquires the acoustic signal of the environment in which the autonomous vehicle is located at the current moment; performs multi-scale spatial decomposition on the acoustic signal to obtain multiple sets of decomposition coefficients at different scales; extracts acoustic features from each set of decomposition coefficients to obtain multiple feature vectors with the same number of sets as the decomposition coefficients; inputs each feature vector into multiple trained neural network models to output classification results for identifying sound-emitting objects in the environment at different scales; and determines whether the sound-emitting object is a preset type of vehicle based on the classification results at all scales. This method, by performing multi-resolution analysis of external acoustic signals and further identifying sound-emitting objects through neural network models, can more accurately determine whether the sound-emitting object is a specific vehicle of interest, thereby providing more accurate environmental information and basis for autonomous vehicles to perform operations such as automatic obstacle avoidance. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram of the structure of an unmanned vehicle according to an embodiment of this application is shown;

[0042] Figure 2 A flowchart of an acoustic detection method in an unmanned driving environment according to an embodiment of this application is shown;

[0043] Figure 3This paper illustrates a system architecture diagram of an acoustic detection method in an unmanned driving environment according to an embodiment of this application.

[0044] Figure 4 A flowchart illustrating the training process of multiple neural network models for an acoustic detection method in an unmanned driving environment according to an embodiment of this application is shown.

[0045] Figure 5a and Figure 5b The composition structure of the deep restricted Zigman machine according to the embodiments of this application and the network structure diagram of the deep restricted Zigman machine when the number of hidden layers of the RNN is 1 are shown respectively.

[0046] Figure 6 A first flowchart of an embodiment of the driverless vehicle control method of this application is shown;

[0047] Figure 7 This paper shows a graph illustrating the relationship between the training sample labels and the straight-line distance of the sound source acquisition system in the unmanned vehicle control method of this application embodiment;

[0048] Figure 8 A second flowchart of an embodiment of the driverless vehicle control method of this application is shown;

[0049] Figure 9 This paper shows a schematic diagram of the acoustic detection device in an unmanned driving environment according to an embodiment of the present application;

[0050] Figure 10 A schematic diagram of the structure of the unmanned vehicle control device according to an embodiment of this application is shown. Detailed Implementation

[0051] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0052] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0053] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0054] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0055] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0056] Please refer to Figure 1 This is a schematic diagram of the structure of an unmanned vehicle proposed in an embodiment of this application. Exemplarily, the unmanned vehicle mainly includes a processor 11, a memory 12, and a sound sensing unit 13. Both the memory 12 and the sound sensing unit 13 are connected to the processor 11. The memory 12 stores a corresponding computer program, and the processor 11 executes the computer program to implement the acoustic detection method or unmanned vehicle control method in the unmanned driving environment of this application embodiment. By performing multi-resolution analysis of external acoustic signals and further identifying the sound-emitting object through a neural network model, it is possible to more accurately determine whether the sound-emitting object is a specific vehicle of interest, thereby providing more accurate environmental information and basis for the unmanned vehicle to perform operations such as automatic obstacle avoidance.

[0057] The processor 11 can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0058] The memory 12 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0059] The sound sensing unit 13 is primarily used to collect ambient sound information around the autonomous vehicle and transmit it to the vehicle control system, enabling the system to perform corresponding operations based on this information. For example, the sound sensing unit 13 may include, but is not limited to, sound sensors located inside and / or outside the vehicle, such as microphones (MICs) or microphone arrays. It should be understood that there may be more than one sound sensing unit in this embodiment. When multiple units are present, synchronous sampling can be performed to obtain sound signals from different directions of the vehicle at the same time, allowing the vehicle to determine the location and distance of the same sound source.

[0060] Based on the above-described structure of the autonomous vehicle, this application proposes an acoustic detection method for autonomous driving environments. The method is described in detail below.

[0061] Please refer to Figure 2 As an example, the acoustic detection method in an unmanned driving environment includes steps S110 to S150:

[0062] S110 acquires the acoustic signals of the environment in which the unmanned vehicle is located at the current moment.

[0063] For example, during the operation of an autonomous vehicle, the acoustic signals of the environment around the vehicle can be collected in real time by the sound sensing unit 13 installed on the vehicle. The vehicle's control system can then perform acoustic analysis on the signals to obtain relevant external signals.

[0064] S120, perform multi-scale spatial decomposition on the acoustic signal to obtain multiple sets of decomposition coefficients of the acoustic signal at different scales.

[0065] Multi-scale spatial decomposition, also known as multi-resolution decomposition, refers to decomposing a signal at different scales to obtain signal expansion coefficients at different scales (resolutions). It can be understood that the original signal before decomposition can be recovered from these wavelet transform expansion coefficients. Based on these expansion results at different scales, target frequencies can be observed and analyzed with varying levels of precision.

[0066] For autonomous vehicles, the acoustic signals they collect from the external environment are often quite complex, typically including sounds of different or the same frequency emitted by various objects. These sounds may be superimposed or masked at certain times, increasing the difficulty of identifying the sound-emitting objects. To address this, this application proposes using different scaling functions to decompose the acoustic signal into sub-bands to obtain non-overlapping sub-bands, thereby extracting features and further classifying the sound-emitting objects from these sub-bands.

[0067] For example, in one implementation, the acoustic signal f acquired at time t can be... (t) Binary wavelet decomposition based on frequency yields multiple sets of wavelet coefficients and a set of residual coefficients at different scales. The P sets of wavelet coefficients can be expressed as... j represents the j-th scale space; a set of residual coefficients can be expressed as Therefore, the multiple sets of wavelet coefficients and a set of residual coefficients These together constitute multiple sets of decomposition coefficients for the acoustic signal. It is understood that the aforementioned binary wavelet decomposition can be implemented using the Mallat algorithm. For details on how the Mallat algorithm calculates the residual coefficients and wavelet coefficients at each scale space, please refer to existing literature; it will not be elaborated upon here.

[0068] Therefore, by performing multi-resolution analysis on the acoustic signal, multiple sets of decomposition coefficients can be obtained, such as... Figure 3As shown, the above-mentioned P+1 sets of decomposition coefficients are included, and then feature extraction is performed on each set of decomposition coefficients to obtain frequency feature information at different resolutions.

[0069] S130, perform acoustic feature extraction on each set of decomposition coefficients to obtain multiple feature vectors with the same number of sets as the decomposition coefficients.

[0070] The aforementioned acoustic features may include, but are not limited to, signal intensity, loudness, Mel-frequency cepstral coefficients, line spectrum pairs, and energy in different frequency bands. Furthermore, the features extracted from the same set of decomposed coefficients are statistically calculated and cascaded to obtain the corresponding feature vector at that scale space. The aforementioned statistical calculation functions may include mean, variance, linear regression coefficients, standard deviation, kurtosis, slope, quartiles, and spacing.

[0071] An audio signal is typically a non-stationary signal. A segment of the signal may include multiple sentences, and a sentence may include multiple frames. When performing feature extraction, features at the frame level, segment level, or speech level can be selected according to actual needs. For example, in one implementation, for the feature extraction in step S130 above, frame-level acoustic feature extraction can be performed to obtain the corresponding features.

[0072] S140 inputs each feature vector into multiple trained neural network models respectively, and outputs the classification results of sound-emitting objects in the environment at different scales.

[0073] In autonomous driving environments, acoustic event detection typically involves two parts: feature learning and pattern recognition. Existing acoustic event detection methods often process these two parts independently. For example, in pattern recognition, methods such as Support Vector Machines (SVM) and Gaussian Mixture Models (GMM) are commonly used. However, the acoustic features used for the detection and recognition task do not effectively utilize the correlation of the recognition task. Therefore, this application proposes an acoustic event detection method based on a combination of wavelet decomposition and depth-restricted Boltzmann machines. This method combines feature learning and pattern recognition in acoustic event detection, thereby fully utilizing the correlation between acoustic features and the recognition task to obtain more accurate external environmental signals.

[0074] Exemplary examples show that each of the above neural network models has the same structure, and will be constructed using a Conditional Restricted Boltzmann Machine (CRBM) as both the input and output layers. A CRBM is a type of stochastic neural network that typically consists of a visible layer and a hidden layer. Neurons within the same layer are independent, while neurons in different layers are interconnected. During training, network parameters such as connection weights and bias vectors between the visible and hidden layers need to be calculated. A Recurrent Neural Network (RNN) is used as the intermediate layer connecting the input and output layers.

[0075] Understandably, in autonomous driving environments, acoustic detection of vehicles is a highly challenging task due to the influence of environmental noise. Compared to environmental noise, vehicle acoustic signals exhibit better temporal stability. Therefore, we employ an RNN as an intermediate layer of a deep restricted Boltzmann machine to perform temporal modeling of the transformation relationship between high-order feature information of feature vectors and label vectors, thereby improving the accuracy of vehicle acoustic signal recognition.

[0076] In one implementation, the aforementioned multiple neural network models can be obtained through pre-construction and training, such as... Figure 4 As shown, its training process may include the following sub-steps S141 to S145:

[0077] S141, P+1 feature vectors with classification labels are used as the first training samples and input into P+1 first conditionally restricted Boltzmann machines respectively. The P+1 first conditionally restricted Boltzmann machines are trained unsupervised using a preset learning algorithm to obtain the network parameters of each first conditionally restricted Boltzmann machine; where P is an integer, and the P+1 feature vectors are obtained by feature extraction and statistical calculation of P sets of wavelet coefficients and 1 set of residual coefficients.

[0078] The aforementioned classification labels can be pre-labeled on the collected training samples, primarily indicating the category of the sound-emitting object. For autonomous vehicles, considering the need for timely avoidance of special vehicles performing tasks, the classification labels will be related to these special vehicles. For example, the classification labels may include, but are not limited to, marking that the acoustic signal contains special vehicles such as police cars, fire trucks, and ambulances. It can be understood that these classification labels correspond to the preset vehicle types mentioned later; that is, to identify a vehicle as a preset type, training samples containing specific frequency signals emitted by that preset type vehicle should be used for classification labeling and training.

[0079] Since the first conditionally constrained Boltzmann machine is designed for feature vectors, it will be referred to as feature CRBM for short. In training the feature CRBM, the aforementioned preset learning algorithms may include, but are not limited to, contrastive divergence (CD) and simulated annealing algorithms. No specific limitation is imposed here; the appropriate algorithm can be selected based on actual needs.

[0080] Suppose the p-th eigenvector is denoted as Where p = 1, 2, ..., P+1, and t represents the t-th time. Taking the contrastive divergence method as an example, for step S141, the maximum likelihood estimation principle can be used as a model. The contrastive divergence method is used to perform unsupervised training on the P+1 feature CRBMs respectively, resulting in P+1 sets of parameters, which are denoted as . in, For the visible layer nodes in the p-th feature CRBM and hidden layer nodes The directed connection weight matrix, For the visible layer nodes in the p-th feature CRBM and hidden layer nodes The directed connection weight matrix, For the visible layer node at time t in the p-th feature CRBM and the visible layer nodes at time t-1 The connection weight matrix.

[0081] It is understood that there is a one-to-one correspondence between the nodes of the feature CRBM and the dimension of the feature vector. In this embodiment, a Conditionally Constrained Boltzmann Machine (CBC) is used to identify the sound-producing object by applying a CBC to the decomposition coefficients at different scales, and then the classification results at all scales are combined to determine the final classification result, thereby improving the recognition accuracy.

[0082] S142, the label vectors corresponding to the P+1 feature vectors are used as the second training samples and respectively input into the P+1 second conditionally restricted Boltzmann machines. The P+1 second conditionally restricted Boltzmann machines are then trained unsupervised using a preset learning algorithm to obtain the network parameters of each second conditionally restricted Boltzmann machine.

[0083] It is understandable that the first conditionally restricted Boltzmann machine mentioned above mainly targets feature vectors, while the second conditionally restricted Boltzmann machine mainly targets label vectors (hereinafter referred to as label CRBM). The terms "first" and "second" are only used to distinguish conditionally restricted Boltzmann machines with different vectors.

[0084] Similarly, for step S142, unsupervised training of the label CRBM can be performed by referring to the above-described feature vector training method. For example, based on the maximum likelihood estimation principle... Unsupervised training of the label CRBM was performed using the contrastive divergence method to obtain the corresponding network parameters. in For the p-th label CRBM visible layer node and hidden layer vectors The directed connection weight matrix, For the p-th label CRBM visible layer vector and hidden layer vectors The directed connection weight matrix, For the p-th label CRBM visible layer vector and The directed connection weight matrix.

[0085] S143, obtain the high-order feature information of each feature vector in the P+1 feature vectors, and the high-order feature information of each label vector in the P+1 label vectors.

[0086] For the p-th feature CRBM, the feature vector at time t is known. and the eigenvectors at time t-1 The eigenvectors can be obtained using the mean field approximation. and High-order feature information

[0087]

[0088] Where s is the sigmoid function; It is the bias vector of the visible layer nodes and hidden layer nodes in the feature CRBM.

[0089] Similarly, for the p-th label CRBM, the label vector at time t is known. and the label vector at time t-1 The label vector can be obtained using the mean field approximation. and High-order feature information

[0090]

[0091] in, It is the bias vector of the visible layer nodes and hidden layer nodes in the label CRBM.

[0092] It is understood that the execution order of the above steps S141, S142 and S143 is not limited. They can be executed simultaneously or in a set order.

[0093] S144. The higher-order feature information of the feature vector and the label vector are used as the input and output of the recurrent neural network, respectively, for supervised training to obtain the network parameters of the recurrent neural network.

[0094] The recurrent neural network (RNN) mentioned above is mainly used to learn and train higher-order feature information from feature vectors to label vectors. As an example, the output of the feature CRBM is... Input of CRBM label These are used as the input and output of the RNN, respectively, for supervised training. In one implementation, the RNN is used as an intermediate layer of a depth-restricted Boltzmann machine, and its objective function can be:

[0095]

[0096] in, and These are the weights of the recurrent neural network. and These are the bias values, λ and λ, respectively. p This is the regularization factor. When the RNN has only one hidden layer... The expression is as follows:

[0097]

[0098] S145, the first conditionally restricted Boltzmann machine, the recurrent neural network, and the second conditionally restricted Boltzmann machine with obtained network parameters are stacked from bottom to top to build a depth-restricted Boltzmann machine.

[0099] Exemplary, after obtaining the network parameters of the feature CRBM, RNN, and label CRBM, as follows: Figure 5a As shown, the trained feature CRBM, RNN, and label CRBM are stacked sequentially from bottom to top to achieve the mapping from acoustic features to object labels. For example, Figure 5b Specifically, the network structure diagram of the depth-restricted Boltzmann machine is given when the number of hidden layers of the RNN is 1 and p=1.

[0100] Therefore, for a given feature vector and For the output of label CRBM Then we have:

[0101]

[0102] Where N represents the normal distribution function, This represents the visible layer node of the p-th label CRBM. and hidden layer vectors The directed connection weight matrix, This represents the bias of the CRBM for the p-th label. This represents the variance of the normal distribution N function. The above formula can be accurately estimated using the least squares method, and the unknown parameters can be solved using an alternating iterative method. It can be understood that the training process for each neural network model is the same.

[0103] For step S140 above, as described above Figure 3 As shown, in the actual detection process, it is only necessary to input the multiple feature vectors extracted at the corresponding time points into these neural network models respectively, and then the corresponding classification label vectors can be output, that is, the classification result of the sound-producing object recognition can be obtained.

[0104] Step S150: Determine whether the sound-emitting object is a preset type of vehicle based on the classification results across all scale spaces.

[0105] In one implementation, step S150 includes: taking a weighted average of the classification results across all scale spaces to obtain a combined classification result. Then, selecting the vehicle type corresponding to the maximum value from the combined classification result to determine whether the sound-emitting object is a preset type of vehicle.

[0106] Since each depth-restricted Boltzmann machine outputs a corresponding predicted classification vector, including the probability of identifying the sound-emitting object as different types of vehicles, these types of vehicles are all designated vehicles of interest, including but not limited to police cars, fire trucks, and ambulances. The numerical range of each label vector component is [0,1]. Therefore, the multiple outputs of these depth-restricted Boltzmann machines can be weighted and averaged to obtain the combined label output for the sound-emitting object. Different weight values ​​can be assigned to depth-restricted Boltzmann machines at different scales; the specific weight values ​​can be set according to the actual situation and are not limited here.

[0107] For example, a weighted average of the probabilities of vehicles belonging to the same type in each depth-constrained Boltzmann machine can be used to obtain a combined average label vector. The vector with the highest probability can then be selected as the final classification result. For instance, assuming the preset vehicle type is vehicle A, if the probability value of vehicle A is the highest in the combined classification result, then the sounding object is confirmed to be a vehicle of the preset type. Furthermore, if it is determined to be an ambulance, fire truck, etc., the autonomous vehicle can further determine whether to perform avoidance maneuvers or other related operations.

[0108] In this embodiment, by performing CRBM unsupervised training on feature vectors and corresponding label vectors at different scales, and extracting high-order feature information of the two types of vectors, and then using the high-order feature information of the feature vectors and label vectors as the input and output of the RNN, the accuracy of vehicle acoustic feature recognition can be improved.

[0109] Please refer to Figure 6 Based on the above-mentioned acoustic detection method in an unmanned driving environment, this application also proposes an unmanned vehicle control method to perform corresponding control operations when the sound-emitting object in the environment is determined to be a preset type of vehicle.

[0110] Examplely, the autonomous vehicle control method includes steps S210 to S230:

[0111] Step S210: The acoustic detection method in the unmanned driving environment described in the above embodiment is used to determine whether the sound-emitting object in the environment where the unmanned vehicle is located is a preset type of vehicle.

[0112] For example, during the operation of an autonomous vehicle, the sound signals in its environment can be acquired in real time, and the acoustic detection method described above can be used to determine whether a specific vehicle, such as a police car or an ambulance, exists. If it does, step S220 is executed.

[0113] Step S220: When it is determined that the sound-emitting object is a preset type of vehicle, the distance between the preset type of vehicle and the unmanned vehicle is calculated based on the acoustic signal acquired in real time.

[0114] Regarding vehicle location information, in one implementation, a subspace-based sound source localization method can be used, while distance information is determined using the range of label value variations. For example, when building training samples, a method such as... Figure 7 The step function shown is used to label the training samples, primarily collecting straight-line distances. During labeling, the sound source's azimuth can vary from 0° to 360°, with intervals of 10°.

[0115] Step S230: When a vehicle of the preset type is detected moving within a preset distance range directly behind the autonomous vehicle, an avoidance operation is performed. For example, if a specific vehicle performing a task is detected moving directly behind the autonomous vehicle and the two are relatively close, an avoidance operation is immediately initiated to improve the intelligence of the autonomous driving system.

[0116] As an alternative, in some cases, a specific sound from a particular vehicle may be detected at a relatively far distance, but it may disappear quickly, such as when the vehicle enters another corner area. In this case, the autonomous vehicle does not need to make real-time distance judgments or perform further avoidance operations, which can save resources and improve the driving range.

[0117] Exemplary, such as Figure 8 As shown, when the sound-emitting object is determined to be a vehicle of a preset type, the acoustic detection method in this unmanned driving environment further includes steps S240 to S250:

[0118] S240 continuously acquires multiple acoustic signals from the environment at subsequent consecutive moments.

[0119] S250: When multiple subsequent acoustic signals all contain a specific frequency signal emitted by the preset type of vehicle, the system determines whether the preset type of vehicle is approaching the autonomous vehicle based on the amplitude change of the specific frequency signal. If it is approaching the autonomous vehicle, then step S230 is executed.

[0120] Please refer to Figure 9 This is a schematic diagram of the acoustic detection device 100 in an unmanned driving environment according to an embodiment of this application.

[0121] Exemplary, the acoustic detection device 100 in the autonomous driving environment includes:

[0122] Acquisition module 110 is used to acquire the acoustic signals of the environment in which the unmanned vehicle is located at the current moment;

[0123] The decomposition module 120 is used to perform multi-scale spatial decomposition on the acoustic signal to obtain multiple sets of decomposition coefficients of the acoustic signal at different scales.

[0124] The feature extraction module 130 is used to extract acoustic features from each group of decomposition coefficients to obtain multiple feature vectors with the same number of decomposition coefficient groups.

[0125] The model recognition module 140 is used to input the multiple feature vectors into multiple trained neural network models respectively, and output the classification results of the sound-emitting objects in the environment at different scales.

[0126] The determination module 150 is used to determine whether the sound-emitting object is a preset type of vehicle based on the classification results in all scale spaces.

[0127] It is understood that the device in this embodiment corresponds to the acoustic detection method in the unmanned driving environment of the above embodiment. The options in this method are also applicable to this embodiment, so they will not be described again here.

[0128] Please refer to Figure 10 This is a schematic diagram of the structure of an unmanned vehicle control device 200 according to an embodiment of this application. Exemplarily, the unmanned vehicle control device 200 includes:

[0129] Acoustic detection module 210 is used to employ the acoustic detection method in the unmanned driving environment of the above embodiment to determine whether the sound-emitting object in the environment where the unmanned vehicle is located is a preset type of vehicle.

[0130] The distance acquisition module 220 is used to calculate the distance from the preset type vehicle to the unmanned vehicle based on the acoustic signal acquired in real time when it is determined that the sound-emitting object is a preset type vehicle.

[0131] The avoidance control module 230 is used to perform an avoidance operation when it detects that a vehicle of the preset type has moved to a preset distance range directly behind the vehicle.

[0132] It is understood that the device in this embodiment corresponds to the unmanned vehicle control method in the above embodiment, and the options in the method are also applicable to this embodiment, so they will not be described again here.

[0133] This application also provides a readable storage medium for storing the computer program used in the aforementioned driverless vehicle.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0136] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An acoustic detection method for unmanned driving environments, characterized in that, include: Acquire the acoustic signals of the environment in which the autonomous vehicle is located at the current moment; The acoustic signal is decomposed into multiple scales to obtain multiple sets of decomposition coefficients at different scales. Acoustic features are extracted from each group of decomposition coefficients to obtain multiple feature vectors with the same number of groups of decomposition coefficients. The multiple feature vectors are respectively input into multiple trained neural network models, and the classification results of sound-emitting objects in the environment are output at different scales. Each neural network model is a deep restricted Boltzmann machine built from bottom to top by a first conditionally restricted Boltzmann machine, a recurrent neural network, and a second conditionally restricted Boltzmann machine. The first conditionally restricted Boltzmann machine is used to learn the feature vectors, the recurrent neural network is used to perform temporal modeling of the transformation relationship between the feature vectors and the higher-order feature information of the label vectors corresponding to the feature vectors, and the second conditionally restricted Boltzmann machine is used to output the label vectors corresponding to the feature vectors as the classification results. Based on the classification results across all scales, determine whether the sound-emitting object is a preset type of vehicle.

2. The acoustic detection method in an unmanned driving environment according to claim 1, characterized in that, The multi-scale spatial decomposition of the acoustic signal yields multiple sets of decomposition coefficients at different scales, including: The acoustic signal is subjected to binary wavelet decomposition based on frequency to obtain multiple sets of wavelet coefficients and a set of residual coefficients in different scale spaces; the multiple sets of wavelet coefficients and the set of residual coefficients together constitute multiple sets of decomposition coefficients of the acoustic signal.

3. The acoustic detection method in an unmanned driving environment according to claim 2, characterized in that, The training process of the multiple neural network models includes: P+1 feature vectors with classification labels are used as the first training samples and are respectively input into P+1 first conditionally restricted Boltzmann machines. The P+1 first conditionally restricted Boltzmann machines are trained unsupervised using a preset learning algorithm to obtain the network parameters of each first conditionally restricted Boltzmann machine; where P is an integer, and the P+1 feature vectors are obtained by feature extraction from P sets of wavelet coefficients and 1 set of residual coefficients; The label vectors corresponding to the P+1 feature vectors are used as the second training samples and are respectively input into the P+1 second conditionally restricted Boltzmann machines. The P+1 second conditionally restricted Boltzmann machines are unsupervised trained using a preset learning algorithm to obtain the network parameters of each second conditionally restricted Boltzmann machine. Obtain the high-order feature information of each feature vector in the P+1 feature vectors, and the high-order feature information of each label vector in the P+1 label vectors; The higher-order feature information of the feature vector and the label vector are used as the input and output of the recurrent neural network, respectively, and supervised training is performed to obtain the network parameters of the recurrent neural network. The first conditionally restricted Boltzmann machine, the recurrent neural network, and the second conditionally restricted Boltzmann machine, which have obtained network parameters, are stacked from bottom to top to build a deep restricted Boltzmann machine as a trained neural network model.

4. The acoustic detection method in an unmanned driving environment according to claim 1, characterized in that, Determining whether the sound-emitting object is a preset type of vehicle based on the classification results across all scale spaces includes: The classification results across all scales are weighted and averaged to obtain a combined classification result. The vehicle type corresponding to the maximum value is selected from the combined classification results to determine whether the sound-emitting object is a preset type of vehicle.

5. The acoustic detection method in an unmanned driving environment according to any one of claims 1 to 4, characterized in that, The preset vehicle type is any one of police cars, fire trucks, and ambulances.

6. A method for controlling an unmanned vehicle, characterized in that, include: The acoustic detection method in the unmanned driving environment according to any one of claims 1 to 5 is used to determine whether the sound-emitting object in the environment where the unmanned vehicle is located is a preset type of vehicle. When the sound-emitting object is determined to be a vehicle of a preset type, the distance from the preset type vehicle to the driverless vehicle is calculated based on the acoustic signals acquired in real time. When a vehicle of the preset type is detected to be moving within a preset distance range directly behind the vehicle, an avoidance operation is performed.

7. The unmanned vehicle control method according to claim 6, characterized in that, When determining that the sound-emitting object is a vehicle of a preset type, the method further includes: Continuously acquire multiple acoustic signals from the environment at subsequent consecutive moments; When multiple subsequent acoustic signals are detected to contain a specific frequency signal emitted by the preset type of vehicle, the amplitude change of the specific frequency signal is used to determine whether the preset type of vehicle is approaching the driverless vehicle. If approaching the driverless vehicle, the distance between the preset type of vehicle and the driverless vehicle is calculated based on the acoustic signals acquired in real time.

8. An acoustic detection device for unmanned driving environments, characterized in that, include: The acquisition module is used to acquire the acoustic signals of the environment in which the autonomous vehicle is located at the current moment; The decomposition module is used to perform multi-scale spatial decomposition on the acoustic signal to obtain multiple sets of decomposition coefficients of the acoustic signal at different scales. The feature extraction module is used to extract acoustic features from each group of decomposition coefficients to obtain multiple feature vectors with the same number of decomposition coefficient groups. The model recognition module is used to input the multiple feature vectors into multiple trained neural network models respectively, and output the classification results of the sound-emitting objects in the environment at different scales. Each neural network model is a deep restricted Boltzmann machine (DRM) built from bottom to top, consisting of a first conditionally restricted Boltzmann machine, a recurrent neural network, and a second DRM. The first DRM learns the feature vectors, the recurrent neural network performs temporal modeling of the transformation relationship between the feature vectors and the higher-order feature information of the corresponding label vectors, and the second DRM outputs the label vectors corresponding to the feature vectors as the classification results. The determination module is used to determine whether the sound-emitting object is a preset type of vehicle based on the classification results in all scale spaces.

9. An unmanned vehicle, characterized in that, The unmanned vehicle includes a sound sensing unit, a processor, and a memory. The sound sensing unit is used to acquire environmental acoustic signals, the memory stores a computer program, and the processor is used to execute the computer program to implement the acoustic detection method in the unmanned driving environment as described in any one of claims 1-5 or the unmanned vehicle control method as described in any one of claims 6-7.

10. A readable storage medium, characterized in that, It stores a computer program, which, when executed on a processor, implements the acoustic detection method in an unmanned driving environment according to any one of claims 1-5 or the unmanned vehicle control method according to any one of claims 6 or 7.

Images