Method and device for target identification based on audio and laser radar point cloud

Abstract

The embodiment of the invention relates to a method and a device for target identification based on audio and laser radar point cloud. The method comprises the following steps: carrying out audio acquisition on a current environment based on a first directional microphone array to obtain a corresponding first array audio set; performing radar scanning on the current environment based on the first laser radar in the audio acquisition process to obtain a corresponding first environment point cloud; performing sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set; performing point cloud target recognition processing according to the first environment point cloud to obtain a corresponding second recognition target set; and performing target type correction on the second recognition target set based on the first recognition target set. According to the invention, the point cloud target recognition result can be effectively corrected based on the audio target recognition result in an environment with bad weather or poor visibility.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for target recognition based on audio and laser radar point cloud. Background Art

[0002] The sensor combination of camera + lidar is a common target recognition sensor combination. The target recognition process of this sensor combination is: first perform target recognition based on the lidar point cloud and camera image, and then use the image-based target recognition result to correct the point cloud target recognition result. However, in practical applications, we found that this conventional sensor combination has a limitation: in bad weather or poor visibility, it is difficult for the camera to capture a clear image for target recognition, that is, in this case, the visual target recognition result cannot effectively correct the point cloud target recognition result. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device, electronic device and computer-readable storage medium for target recognition based on audio and laser radar point cloud in response to the defects of the prior art. The present invention provides a sensor combination of a directional microphone array + laser radar, and the directional microphone array of the combination performs audio collection on the current environment to obtain a corresponding microphone audio set, and the laser radar of the combination performs radar scanning on the current environment to obtain a corresponding environmental point cloud; and performs sound source target recognition processing based on the microphone audio set to obtain a corresponding first recognition target set, and performs point cloud target recognition processing based on the environmental point cloud to obtain a corresponding second recognition target set; and uses the first recognition target set to correct the target type of the second recognition target set. The directional microphone array + laser radar sensor combination provided by the present invention can be used as a supplementary solution to the traditional camera + laser radar combination. The sensor combination of the present invention can effectively correct the point cloud target recognition results based on the audio target recognition results in environments with bad weather or poor visibility.

[0004] To achieve the above object, a first aspect of an embodiment of the present invention provides a method for target recognition based on audio and laser radar point cloud, the method comprising:

[0005] Based on the first directional microphone array, audio of the current environment is collected to obtain a corresponding first array audio set; and during the audio collection process, radar scanning of the current environment is performed based on the first laser radar to obtain a corresponding first environment point cloud;

[0006] Performing sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set; the first recognition target set includes one or more first recognition targets; each of the first recognition targets includes first target coordinates and a first target type; the first target types include men, women, children, animals, vehicles and others;

[0007] Performing point cloud target recognition processing according to the first environmental point cloud to obtain a corresponding second recognition target set; the second recognition target set includes one or more second recognition targets; each of the second recognition targets includes second target coordinates, second target type, second target shape, and second target orientation; the second target types include men, women, children, animals, vehicles, and others;

[0008] The target type of the second recognition target set is corrected based on the first recognition target set.

[0009] Preferably, the first directional microphone array includes a plurality of first directional microphones M i , 1≤index i≤N ini , N ini is the total number of directional microphones in the first directional microphone array; each of the first directional microphones M i Corresponding to a microphone coordinate (x, y, z), each of the microphone coordinates (x, y, z) is a world coordinate system coordinate;

[0010] The first array audio set includes a plurality of first microphone audio X i ; The first microphone audio X i With the first directional microphone M i One-to-one correspondence; each of the first microphone audio X i is the first directional microphone M corresponding to i The ambient audio obtained by collecting audio from the current environment; all the first directional microphones M i When audio is collected from the current environment, it is collected synchronously, that is, all the first microphone audio X i The collection start time, collection end time and total collection duration are the same.

[0011] Preferably, performing sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set specifically includes:

[0012] Based on the sound source separation model, the first microphone audio X i Perform sound source audio separation processing to obtain a corresponding first sound source audio set; and count the total number of sound source audios in each of the first sound source audio sets to obtain a corresponding first sound source total number ni ; The sound source separation model includes at least a TasNet model, a Conv-TasNet model, a Multi-Stream Conv-TasNet model, a Wave-U-Net model, a MulCat DPRNN model and a DPTNet model; The first sound source audio set includes one or more first sound source audio x i,j , 1≤indexj≤n i Each of the first sound source audio x i,j The length of time corresponding to the first microphone audio X i The length of time is consistent;

[0013] For each of the first sound source audio x i,j Perform sound source feature extraction to obtain the corresponding first feature vector a i,j ; The first feature vector a i,j Including start time feature, end time feature, Mel frequency cepstral coefficient feature vector, short time energy feature vector and spectrum envelope feature vector;

[0014] Based on the audio classification model, each of the first sound source audio x i,j Perform sound source classification and recognition processing to obtain the corresponding first sound source type b i,j ; The audio classification model includes at least Wav2vec 2.0 model, HuBERT model and VGGish model; The first sound source type b i,j including men, women, children, animals, vehicles and others;

[0015] The total number of the first sound sources n i The maximum value of the corresponding maximum total number of sound sources; and any of the first total number of sound sources n i The first sound source audio set that matches the maximum total number of sound sources is used as the corresponding basic sound source audio set; and all other first sound source audio sets except the basic sound source audio set are recorded as the corresponding comparison sound source audio sets;

[0016] For all the first sound source audio x in the basic sound source audio set i,j The homologous target audio in all the compared sound source audio sets is retrieved and clustered to obtain a corresponding first target set; the first target set includes one or more first target audio sets; the number of the first target audio sets is the same as the number of the first sound source audio x in the basic sound source audio set. i,j The same number; each of the first target audio sets includes a plurality of the first sound source audio x i,j ; All the first sound source audio x in each of the first target audio sets i,jAll of the first sound source audio x in each of the first target audio sets correspond to the same sound source target. i,j The corresponding first sound source type b i,j All the same;

[0017] Based on each of the first target audio sets, the sound source target positioning estimation is performed to obtain the corresponding first target coordinates; and any one of the first sound source audio x in each of the first target audio sets is i,j The corresponding first sound source type b i,j extracting it as the corresponding first target type;

[0018] The first target coordinates and the first target type corresponding to each of the first target audio sets form a corresponding first recognition target; and all the obtained first recognition targets form a corresponding first recognition target set.

[0019] Furthermore, the first sound source audio x i,j Perform sound source feature extraction to obtain the corresponding first feature vector a i,j , specifically including:

[0020] The first sound source audio x is sampled at a preset first audio sampling frequency. i,j Performing audio signal sampling processing to obtain a corresponding first signal sampling sequence; the first signal sampling sequence is formed by sequentially sorting a plurality of first signal sampling points; each of the first signal sampling points includes a first sampling point time and a first sampling point amplitude;

[0021] The first and last first signal sampling points of the first signal sampling sequence are recorded as the corresponding starting sampling point and ending sampling point; and the only first signal sampling point in the first signal sampling sequence whose amplitudes of all the first sampling points between the starting sampling point and the first signal sampling point are 0 is recorded as the corresponding valid signal starting point, and the time interval between the valid signal starting point and the starting sampling point is calculated to obtain the corresponding starting time interval △t s ; and the only first signal sampling point in the first signal sampling sequence whose amplitudes of all the first sampling points between the end sampling point and the end sampling point are 0 is recorded as the corresponding valid signal end point, and the time interval between the end sampling point and the valid signal end point is calculated to obtain the corresponding end time interval △t e ; and count the total number of the first signal sampling points of the first signal sampling sequence and use the statistical result as the corresponding first sequence length L; and based on the starting time interval △t s , the end time interval Δt eThe starting time feature and the ending time feature corresponding to the first sequence length L are set, and the starting time feature = Δt s / L, end time characteristic = 1-△t e / L;

[0022] The first signal sampling sequence is subjected to signal framing processing according to a preset audio framing rule to obtain a corresponding first signal frame sequence; the first signal frame sequence is formed by sequentially sorting one or more first signal frames; each of the first signal frames is formed by sequentially sorting a plurality of first signal sampling points; the number of the first signal sampling points of all the first signal frames is the same;

[0023] Identify the Mel-frequency cepstral coefficient feature vectors of each of the first signal frames to obtain a corresponding first frame coefficient feature vector; and sequentially sort all the obtained first frame coefficient feature vectors to form the corresponding Mel-frequency cepstral coefficient feature vector;

[0024] Calculate the signal energy of each of the first signal frames and use the calculation result as the corresponding first frame signal energy; and sort all the obtained first frame signal energies in order to form the corresponding short-time energy feature vector;

[0025] Identifying the spectrum envelope feature vectors of each of the first signal frames to obtain a corresponding first frame envelope feature vector; and sequentially sorting all the obtained first frame envelope feature vectors to form a corresponding spectrum envelope feature vector;

[0026] The first sound source audio x i,j The corresponding start time feature, the end time feature, the Mel frequency cepstral coefficient feature vector, the short-time energy feature vector and the spectrum envelope feature vector are sequentially arranged to form the corresponding first feature vector a i,j .

[0027] Furthermore, the first sound source audio x of all the first sound source audios in the basic sound source audio set i,j Performing a search and clustering process on the homologous target audios in all the compared sound source audio sets to obtain a corresponding first target full set specifically includes:

[0028] Step 51: The first audio source x of the basic audio source set is i,j As the corresponding current sound source audio;

[0029] Step 52: convert the first feature vector a corresponding to the current sound source audio i,j and the first sound source type b i,j As the corresponding current feature vector and current sound source type;

[0030] Step 53, perform a round of traversal on all the compared sound source audio sets; and in this round of traversal, use the currently traversed compared sound source audio set as the corresponding current audio set; and add the first sound source type b in the current audio set. i,j The first sound source audio x that matches the current sound source type i,j and when the number of the first preliminary selected audios is not 0, the first feature vector a corresponding to each of the first preliminary selected audios is i,j The first similarity is calculated with the similarity of the current feature vector; and the largest of the first similarities is used as the corresponding maximum similarity; and when the maximum similarity exceeds a preset similarity threshold, the first preliminary selected audio corresponding to the maximum similarity is used as a homologous target audio matching the current sound source audio; and at the end of this round of traversal, the current sound source audio and all the homologous target audios obtained in this round of traversal are grouped into one category to form a corresponding first target audio set;

[0031] Step 54: Check whether the current sound source audio is the last of the first sound source audio x in the basic sound source audio set. i,j If not, the next first sound source audio x of the basic sound source audio set i,j Extract it as the new current sound source audio, and return to step 52; if so, all the first target audio sets obtained form the corresponding first target complete set.

[0032] Further, the performing sound source target positioning estimation based on each of the first target audio sets to obtain the corresponding first target coordinates specifically includes:

[0033] Step 61: taking each of the first target audio sets as the corresponding current target audio set; and performing a calculation on the first sound source audio x of the current target audio set. i,j The total number of the current audio is counted to obtain the corresponding total number G of the current audio; and the sound source target corresponding to the current target audio set is recorded as the corresponding current sound source target; and the unknown coordinates of the current sound source target are set to the corresponding sound source coordinate variable (x * ,y * ,z * );

[0034] Step 62: any two of the first sound source audios x in the current target audio set are combined into i,j Denoted as the corresponding microphone audio v p and microphone audio v qForm a corresponding microphone audio pair, 1≤index p≤G, 1≤index q≤G, p≠q; and convert the microphone audio v p 、v q The corresponding microphone coordinates (x, y, z) are recorded as the corresponding microphone coordinates (x p ,y p ,z p ) and microphone coordinates (x q ,y q ,z q ); and the audio v p 、v q The first sampling point time corresponding to the starting point of the effective signal is recorded as the corresponding microphone arrival time t p and microphone arrival time t q ; and based on the microphone arrival time t p ,t q Calculate the corresponding microphone arrival time difference △t pq =t p -t q ; and the microphone coordinates (x p ,y p ,z p ), the microphone coordinates (x q ,y q ,z q ) and the microphone arrival time difference △t pq forming a corresponding microphone pair parameter group;

[0035] Step 63: According to the TDOA algorithm, all the microphone pair parameter groups and the sound source coordinate variables (x * ,y * ,z * ) construct the corresponding first set of equations;

[0036] The first set of equations is: c×Δt pq =r p -r q ,

[0037] in,

[0038]

[0039] c is the preset sound wave propagation speed parameter,

[0040] r p is the sound source coordinate variable (x * ,y * ,z * ) to the microphone coordinates (x p ,yp ,z p ),

[0041] r q is the sound source coordinate variable (x * ,y * ,z * ) to the microphone coordinates (x q ,y q ,z q ) is the shortest sound wave propagation distance between two objects;

[0042] Step 64, constructing a corresponding first objective function F based on the first set of equations;

[0043] The first objective function F is:

[0044]

[0045] Step 65: the sound source coordinate variable (x * ,y * ,z * ) is solved and the solution is used as the corresponding first target coordinate.

[0046] Preferably, performing point cloud target recognition processing according to the first environmental point cloud to obtain a corresponding second recognition target set specifically includes:

[0047] The first environment point cloud is subjected to target detection and classification recognition processing based on a point cloud target recognition model to obtain a corresponding second detection set; the point cloud target recognition model includes at least a PointRCNN model, a VoxelNet model, a PointPillars model, a PillarNet model, a SECOND model, and a PV-RCNN model; the second detection set includes one or more first target detection frames; each of the first target detection frames includes a first detection frame shape, a first detection frame orientation, a first detection frame center point coordinate, and a first detection frame type; the first detection frame types include men, women, children, animals, vehicles, and others;

[0048] Performing coordinate conversion on the center point coordinates of the first detection frame of each first target detection frame from the radar coordinate system to the world coordinate system to obtain the corresponding first center point world coordinates;

[0049] A corresponding second recognition target is formed by the world coordinates of the first center point of each first target detection frame, the first detection frame type, the first detection frame shape and the first detection frame orientation as the corresponding second target coordinates, the second target type, the second target shape and the second target orientation; and a corresponding second recognition target set is formed by all the obtained second recognition targets.

[0050] Preferably, the performing target type correction on the second identified target set based on the first identified target set specifically includes:

[0051] A round of traversal is performed on all the second identified targets in the second identified target set; and in this round of traversal, the second identified target currently traversed is taken as the corresponding current target, and the second target coordinates and the second target type of the current target are taken as the corresponding current target coordinates and the current target type; and the straight-line distances between each of the first target coordinates in the first identified target set and the current target coordinates are calculated to obtain the corresponding first distance; and the smallest first distance is taken as the corresponding minimum distance; and when the minimum distance does not exceed the preset minimum distance threshold, the first target type of the first identified target corresponding to the minimum distance is taken as the corresponding current comparison target type; and whether the current comparison target type matches the current target type is identified, and if not, the second target type of the current target is corrected to the corresponding current comparison target type.

[0052] Preferably, the method further comprises:

[0053] Before using the sound source separation model, the sound source separation model is trained based on a preset first data set; wherein the first data set includes multiple first data records; the first data record includes a first training scene audio and a first label separation audio set; the first training scene audio is a scene audio containing one or more first sound sources; the target types of the first sound source include male adults of different ages, female adults of different ages, children of different ages, different kinds of animals, different kinds of vehicles, and other different kinds of sound sources except humans, animals, and vehicles; the first label separation audio set includes one or more first label separation audios; the first label separation audio of the first label separation audio set corresponds one-to-one to the first sound source in the first training scene audio; the audio length of each first label separation audio is the same as the audio length of the corresponding first training scene audio;

[0054] Before using the audio classification model, the audio classification model is trained based on a preset second data set; wherein the second data set includes multiple second data records; the second data record includes a first training sound source audio and a first label type; the first training sound source audio is the audio data of a single sound source target; the first label type is the target type of the corresponding single sound source target, including man, woman, child, animal, vehicle and others; the first label type of man indicates that the corresponding single sound source target is a male adult, the first label type of woman indicates that the corresponding single sound source target is a female adult, the first label type of child indicates that the corresponding single sound source target is an underage child, the first label type of animal indicates that the corresponding single sound source target is a type of non-human animal, the first label type of vehicle indicates that the corresponding single sound source target is a type of vehicle, and the first label type of other indicates that the corresponding single sound source target is a type of sound source other than humans, animals and vehicles.

[0055] Preferably, the method further comprises:

[0056] Before using the point cloud target recognition model, the point cloud target recognition model is trained based on a preset third data set; wherein the third data set includes multiple third data records; the third data record includes a first training point cloud and a first label detection set; the first label detection set includes one or more first label target detection frames; each of the first label target detection frames includes a first label detection frame shape, a first label detection frame orientation, a first label detection frame center point coordinates and a first label detection frame type; the first label detection frame types include men, women, children, animals, vehicles and others; the first label detection frame type is a man, indicating that the corresponding target point cloud in the first label target detection frame is a If the first label detection frame type is woman, the target point cloud in the first label target detection frame is a female adult; if the first label detection frame type is child, the target point cloud in the first label target detection frame is an underage child; if the first label detection frame type is animal, the target point cloud in the first label target detection frame is a type of non-human animal; if the first label detection frame type is vehicle, the target point cloud in the first label target detection frame is a type of vehicle; if the first label detection frame type is other, the target point cloud in the first label target detection frame is a type of target other than humans, animals, and vehicles.

[0057] Preferably, the performing model training on the sound source separation model based on a preset first data set before using the sound source separation model specifically includes:

[0058] Step 111, randomly dividing the first data set into corresponding first training sets and first evaluation sets based on a preset first training evaluation ratio;

[0059] Wherein, both the first training set and the first evaluation set include a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first training evaluation ratio;

[0060] Step 112, extracting the first first data record in the first training set as the corresponding current training record;

[0061] Step 113, inputting the first training scene audio of the current training record into the sound source separation model to perform sound source audio separation processing to obtain a corresponding first training separation audio set;

[0062] Wherein, the first training separated audio set includes one or more first training separated audios;

[0063] Step 114, bringing the first training separated audio set and the first label separated audio set of the current training record into a preset first model loss function to calculate and obtain a corresponding first loss value;

[0064] Wherein, the first model loss function includes an L1 loss function, an L2 loss function, an SDR loss function, an SI-SDR loss function and a spectrum reconstruction error loss function;

[0065] Step 115, identifying whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identifying whether the current training record is the last first data record in the first training set, if so, turning to step 116, if not, extracting the next first data record in the first training set as the new current training record and returning to step 113; if the first loss value does not satisfy the first loss value range, performing a round of model parameter optimization on the sound source separation model in the direction of minimizing the first model loss function based on a preset first model parameter optimizer, and returning to step 113 at the end of this round of parameter optimization;

[0066] Wherein, the first model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0067] Step 116, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the first training scene audio of the current evaluation record into the sound source separation model for sound source audio separation processing to obtain the corresponding second training separation audio set; and form a corresponding first prediction label pair by the second training separation audio set and the first label separation audio set of the current evaluation record; and at the end of this round of traversal, bring all the first prediction label pairs obtained into the preset first model evaluation function for calculation to obtain the corresponding first evaluation value;

[0068] Wherein, the first model evaluation function at least includes an MSE error function and an RMSE error function;

[0069] Step 117, identifying whether the first evaluation value satisfies a preset first evaluation value range; if the first evaluation value does not satisfy the first evaluation value range, returning to step 111; if the first evaluation value satisfies the first evaluation value range, solidifying the model parameters of the sound source separation model and confirming that the current model training of the sound source separation model is completed.

[0070] Preferably, the performing model training on the audio classification model based on a preset second data set before using the audio classification model specifically includes:

[0071] Step 121, randomly dividing the second data set into corresponding second training sets and second evaluation sets based on a preset second training evaluation ratio;

[0072] Wherein, both the second training set and the second evaluation set include a plurality of second data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second training evaluation ratio;

[0073] Step 122, extracting the first second data record in the second training set as the corresponding current training record;

[0074] Step 123, inputting the first training sound source audio of the current training record into the audio classification model to perform sound source classification and recognition processing to obtain a corresponding first training sound source type;

[0075] Wherein, the first training sound source types include men, women, children, animals, vehicles and others;

[0076] Step 124, bringing the first training sound source type and the first label type of the current training record into a preset second model loss function to calculate and obtain a corresponding second loss value;

[0077] Wherein, the second model loss function includes a multi-classification cross entropy loss function;

[0078] Step 125, identifying whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, identifying whether the current training record is the last second data record in the second training set, if so, turning to step 126, if not, extracting the next second data record in the second training set as the new current training record and returning to step 123; if the second loss value does not satisfy the second loss value range, performing a round of model parameter optimization on the audio classification model in the direction of minimizing the second model loss function based on a preset second model parameter optimizer, and returning to step 123 at the end of this round of parameter optimization;

[0079] Wherein, the second model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0080] Step 126, perform a round of traversal on all the second data records of the second evaluation set; and in this round of traversal, use the second data record currently traversed as the corresponding current evaluation record; and input the first training sound source audio of the current evaluation record into the audio classification model for sound source classification and recognition processing to obtain the corresponding second training sound source type; and form a corresponding second predicted label pair by the second training sound source type and the first label type of the current evaluation record; and at the end of this round of traversal, calculate the precision and recall based on all the obtained second predicted label pairs to obtain the corresponding first precision and first recall; and calculate the F1 score based on the first precision and the first recall to obtain the corresponding first F1 score; and use the first F1 score as the corresponding second evaluation value;

[0081] Step 127, identifying whether the second evaluation value satisfies a preset second evaluation value range; if the second evaluation value does not satisfy the second evaluation value range, returning to step 121; if the second evaluation value satisfies the second evaluation value range, solidifying the model parameters of the audio classification model and confirming that the current model training of the audio classification model is completed.

[0082] Preferably, the step of performing model training on the point cloud target recognition model based on a preset third data set before using the point cloud target recognition model specifically includes:

[0083] Step 131, randomly dividing the third data set into corresponding third training sets and third evaluation sets based on a preset third training evaluation ratio;

[0084] Wherein, the third training set and the third evaluation set both include a plurality of the third data records; the ratio of the total number of records in the third training set to the total number of records in the third evaluation set satisfies the third training evaluation ratio;

[0085] Step 132, extracting the first third data record in the third training set as the corresponding current training record;

[0086] Step 133, inputting the first training point cloud of the current training record into the point cloud target recognition model to perform target detection and classification recognition processing to obtain a corresponding first training detection set;

[0087] The first training detection set includes one or more first training target detection frames; each of the first training target detection frames includes a first training detection frame shape, a first training detection frame orientation, a first training detection frame center point coordinate, and a first training detection frame type; the first training detection frame types include men, women, children, animals, vehicles, and others;

[0088] Step 134, taking each of the first label target detection frames of the first label detection set of the current training record as the corresponding current label frame; and taking the first training target detection frame in the first training detection set that is closest to the current label frame as the corresponding current training frame; and constructing a full zero vector with a vector length of 6 for the current label frame and the current training frame as the corresponding first label type vector and first training type vector, respectively, and establishing a one-to-one correspondence between the 6 vector data of the first label type vector and the first training type vector and the 6 classification types; and forming a corresponding first label detection frame shape, the first label detection frame orientation, and the first label detection frame center point coordinates of the current label frame. a training geometric vector, and a corresponding first label geometric vector is formed by the shape of the first training detection frame, the orientation of the first training detection frame, and the coordinates of the center point of the first training detection frame of the current training frame; and the vector data corresponding to the first label detection frame type of the current label frame in the first label type vector is reset to 1; and the vector data corresponding to the first training detection frame type of the current training frame in the first training type vector is reset to 1; and the first training geometric vector corresponding to the current label frame and the first label geometric vector form a corresponding first geometric vector pair; and the first training type vector corresponding to the current label frame and the first label type vector form a corresponding first type vector pair;

[0089] The six classification types include men, women, children, animals, vehicles and others;

[0090] Step 135, bringing all the first geometric vector pairs corresponding to the current training record into a preset third model loss function to calculate a corresponding third loss value; bringing all the first type vector pairs corresponding to the current training record into a preset fourth model loss function to calculate a corresponding fourth loss value; and performing weighted sum calculation on the third and fourth loss values ​​to obtain a corresponding fifth loss value;

[0091] The third model loss function includes an L1 loss function and an L2 loss function;

[0092] The fourth model loss function includes a multi-classification cross entropy loss function;

[0093] Step 136, identifying whether the fifth loss value satisfies a preset third loss value range; if the fifth loss value satisfies the third loss value range, identifying whether the current training record is the last third data record in the third training set, if so, turning to step 137, if not, extracting the next third data record in the third training set as the new current training record and returning to step 133; if the fifth loss value does not satisfy the third loss value range, performing a round of model parameter optimization on the point cloud target recognition model in the direction of minimizing the third and fourth model loss functions based on a preset third model parameter optimizer, and returning to step 133 at the end of this round of parameter optimization;

[0094] Wherein, the third model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0095] Step 137, inputting the first training point cloud of each of the third data records in the third evaluation set into the point cloud target recognition model for target detection and classification recognition processing to obtain a corresponding second training detection set;

[0096] The second training detection set includes one or more second training target detection frames; each of the second training target detection frames includes a second training detection frame shape, a second training detection frame orientation, a second training detection frame center point coordinate, and a second training detection frame type; the second training detection frame types include men, women, children, animals, vehicles, and others;

[0097] Step 138, taking the first label detection set of each of the third data records in the third evaluation set as the corresponding current label detection set; and taking the second training detection set corresponding to the current label detection set as the corresponding current training detection set; and taking each of the first label target detection frames in the current label detection set as the corresponding current label frame; and forming a corresponding second training geometric vector by the shape of the first label detection frame, the orientation of the first label detection frame, and the coordinates of the center point of the first label detection frame of the current label frame, and forming a corresponding second label geometric vector by the shape of the second training detection frame, the orientation of the second training detection frame, and the coordinates of the center point of the second training detection frame of the current training frame; and forming a corresponding second geometric vector pair by the second training geometric vector corresponding to the current label frame and the second label geometric vector;

[0098] Step 139, taking the first label detection set of each of the third data records of the third evaluation set as the corresponding current label detection set; taking the second training detection set corresponding to the current label detection set as the corresponding current training detection set; taking each of the first label target detection frames in the current label detection set as the corresponding current label frame; taking the second training target detection frame in the current training detection set that is closest to the current label frame as the corresponding current training frame; taking the first label detection frame type of the current label frame as the corresponding current label type, and taking the second training detection frame type of the current training frame as the corresponding current training type; and constructing a full zero vector with a vector length of 6 for the current label frame and the current training frame respectively as the corresponding second label type vector and second training type vector; and establishing a one-to-one correspondence between the 6 vector data of the second label type vector and the second training type vector and the 6 classification types; and resetting the vector data corresponding to the current label type in the second label type vector to 1, and resetting the vector data corresponding to the previous training type in the second training type vector to 1; and forming a corresponding second type vector pair by the second training type vector corresponding to the current label frame and the second label type vector;

[0099] Step 140, bringing all the obtained second geometric vector pairs into a preset second model evaluation function to calculate and obtain a corresponding third evaluation value; and performing precision and recall calculations on all the obtained second type vector pairs to obtain a corresponding second precision and second recall, and performing F1 score calculations based on the second precision and the second recall to obtain a corresponding second F1 score, and using the second F1 score as the corresponding fourth evaluation value;

[0100] Wherein, the second model evaluation function at least includes an MSE error function and an RMSE error function;

[0101] Step 141, and identify the third and fourth evaluation values; if the third evaluation value does not meet the preset third evaluation value range or the fourth evaluation value does not meet the preset fourth evaluation value range, return to step 131; if the third evaluation value meets the preset third evaluation value range and the fourth evaluation value meets the preset fourth evaluation value range, the model parameters of the point cloud target recognition model are solidified once and it is confirmed that the current model training of the point cloud target recognition model is completed.

[0102] A second aspect of an embodiment of the present invention provides a device for implementing the method for target recognition based on audio and laser radar point cloud described in the first aspect, the device comprising: a data acquisition module, an audio target recognition processing module, a point cloud target recognition processing module and a target recognition result correction module;

[0103] The data acquisition module is used to collect audio of the current environment based on the first directional microphone array to obtain a corresponding first array audio set; and to perform radar scanning on the current environment based on the first laser radar to obtain a corresponding first environment point cloud during the audio acquisition process;

[0104] The audio target recognition processing module is used to perform sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set; the first recognition target set includes one or more first recognition targets; each of the first recognition targets includes first target coordinates and a first target type; the first target type includes men, women, children, animals, vehicles and others;

[0105] The point cloud target recognition processing module is used to perform point cloud target recognition processing according to the first environmental point cloud to obtain a corresponding second recognition target set; the second recognition target set includes one or more second recognition targets; each second recognition target includes second target coordinates, second target type, second target shape and second target orientation; the second target type includes men, women, children, animals, vehicles and others;

[0106] The target recognition result correction module is used to correct the target type of the second recognition target set based on the first recognition target set.

[0107] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0108] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0109] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0110] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0111] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for target recognition based on audio and laser radar point cloud. It can be seen from the above invention content that the embodiments of the present invention provide a sensor combination of a directional microphone array + laser radar, and the directional microphone array of the combination performs audio collection on the current environment to obtain a corresponding microphone audio set, and the laser radar of the combination performs radar scanning on the current environment to obtain a corresponding environmental point cloud; and performs sound source target recognition processing based on the microphone audio set to obtain a corresponding first recognition target set, and performs point cloud target recognition processing based on the environmental point cloud to obtain a corresponding second recognition target set; and uses the first recognition target set to correct the target type of the second recognition target set. The sensor combination of a directional microphone array + laser radar provided in the embodiments of the present invention can be used as a supplementary solution to the traditional camera + laser radar combination. The sensor combination of the embodiments of the present invention can effectively correct the point cloud target recognition results based on the audio target recognition results in an environment with bad weather or poor visibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 A schematic diagram of a method for target recognition based on audio and laser radar point cloud provided in Embodiment 1 of the present invention;

[0113] Figure 2 A module structure diagram of a device for target recognition based on audio and laser radar point cloud provided in Embodiment 2 of the present invention;

[0114] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0115] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0116] A method for target recognition based on audio and laser radar point cloud is provided in the first embodiment of the present invention. Figure 1 A method for target recognition based on audio and laser radar point cloud provided in the first embodiment of the present invention is shown in a schematic diagram, which includes the following main steps:

[0117] Step 1: collect audio of the current environment based on the first directional microphone array to obtain a corresponding first array audio set; and during the audio collection process, perform a radar scan on the current environment based on the first laser radar to obtain a corresponding first environment point cloud.

[0118] Here, the first directional microphone array of the embodiment of the present invention includes a plurality of first directional microphones M i , 1≤index i≤N ini , N ini is the total number of directional microphones in the first directional microphone array; each first directional microphone M i Corresponding to a microphone coordinate (x, y, z), each microphone coordinate (x, y, z) is a world coordinate system coordinate.

[0119] The first array audio set of the embodiment of the present invention includes a plurality of first microphone audio X i ;First Microphone Audio X i With the first directional microphone M i One-to-one correspondence; each first microphone audio X i is the corresponding first directional microphone M i The ambient audio obtained by collecting audio from the current environment; all first directional microphones M i When collecting audio in the current environment, all audio from the first microphone X i The collection start time, collection end time and total collection duration are the same.

[0120] Step 2, performing sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set;

[0121] The first identification target set includes one or more first identification targets; each first identification target includes first target coordinates and a first target type; the first target type includes men, women, children, animals, vehicles and others;

[0122] Specifically, step 21 includes: based on the sound source separation model, each first microphone audio X i Perform sound source audio separation processing to obtain a corresponding first sound source audio set; and count the total number of sound source audios in each first sound source audio set to obtain a corresponding first sound source total number n i ;

[0123] Here, the sound source separation model of the embodiment of the present invention is an end-to-end audio separation model, and commonly used ones include at least TasNet model, Conv-TasNet model, Multi-Stream Conv-TasNet model, Wave-U-Net model, MulCat DPRNN model and DPTNet model, etc.; the first sound source audio set of the embodiment of the present invention includes one or more first sound source audio x i,j , 1≤indexj≤n i ; Each first sound source audio x i,j The length of time corresponding to the first microphone audio X i The length of time is consistent;

[0124] Step 22: for each first sound source audio x i,j Perform sound source feature extraction to obtain the corresponding first feature vector a i,j ;

[0125] Among them, the first eigenvector a i,j Including start time feature, end time feature, Mel frequency cepstral coefficient feature vector, short time energy feature vector and spectrum envelope feature vector;

[0126] Specifically, step 221 includes: sampling the first sound source audio x according to the preset first audio sampling frequency. i,j Performing audio signal sampling processing to obtain a corresponding first signal sampling sequence;

[0127] Wherein, the first signal sampling sequence is formed by sequentially sorting a plurality of first signal sampling points; each first signal sampling point includes a first sampling point time and a first sampling point amplitude;

[0128] Here, the first audio sampling frequency is a preset sampling frequency parameter;

[0129] Step 222, record the first and last first signal sampling points of the first signal sampling sequence as the corresponding starting sampling point and ending sampling point; and record the only first signal sampling point in the first signal sampling sequence where all first sampling points between the starting sampling point and the first signal sampling point have an amplitude of 0 as the corresponding valid signal starting point, and calculate the time interval between the valid signal starting point and the starting sampling point to obtain the corresponding starting time interval △t s ; and the first signal sampling point in the first signal sampling sequence where the amplitude of all first sampling points between the end sampling point and the end sampling point is 0 is recorded as the corresponding valid signal end point, and the time interval between the end sampling point and the valid signal end point is calculated to obtain the corresponding end time interval △t e; and count the total number of the first signal sampling points of the first signal sampling sequence and use the statistical result as the corresponding first sequence length L; and based on the starting time interval △t s , end time interval △t e The start time feature and end time feature corresponding to the first sequence length L are set, and the start time feature = △t s / L, end time characteristic = 1-△t e / L;

[0130] Here, the start time feature and the end time feature are actually the start and end positions of the effective signal in the audio signal, that is, the first sound source audio x i,j The relative time position in

[0131] Step 223, performing signal framing processing on the first signal sampling sequence according to a preset audio framing rule to obtain a corresponding first signal frame sequence;

[0132] The first signal frame sequence is formed by sequentially sorting one or more first signal frames; each first signal frame is formed by sequentially sorting a plurality of first signal sampling points; and the number of first signal sampling points of all first signal frames is the same;

[0133] Here, the audio framing rule is a preset sliding framing / slicing rule, corresponding to which are two parameters: sliding step length and single frame / slicing length;

[0134] Framing based on the audio framing rule actually means taking the starting position of the first signal sampling sequence as the first framing point, and extracting a subsequence with a time length of a single frame / slice length backward from the current framing point as the first first signal frame; then taking the sequence index obtained by adding the sequence index of the current framing point to the sliding step as a new framing point, and extracting a subsequence with a time length of a single frame / slice length backward from the current framing point as the next first signal frame; then taking the sequence index obtained by adding the sequence index of the current framing point to the sliding step as a new framing point, and so on, until the latest framing point exceeds the first signal sampling sequence;

[0135] Step 224, identifying the Mel-frequency cepstral coefficient feature vectors of each first signal frame to obtain a corresponding first frame coefficient feature vector; and sequentially sorting all the obtained first frame coefficient feature vectors to form a corresponding Mel-frequency cepstral coefficient feature vector;

[0136] Here, the calculation method for extracting Mel-frequency cepstral coefficient features of audio signals is a conventional public calculation method, which will not be further described here;

[0137] Step 225, calculating the signal energy of each first signal frame and using the calculation result as the corresponding first frame signal energy; and sequentially sorting all the obtained first frame signal energies to form a corresponding short-time energy feature vector;

[0138] Here, calculating the signal energy of the audio signal is also a conventional public calculation method, which will not be further described here;

[0139] Step 226, identifying the spectrum envelope feature vectors of each first signal frame to obtain a corresponding first frame envelope feature vector; and sequentially arranging all the obtained first frame envelope feature vectors to form a corresponding spectrum envelope feature vector;

[0140] Here, the calculation method for extracting the spectrum envelope feature of the audio signal is a conventional public calculation method, which will not be further described here;

[0141] Step 227, the current first sound source audio x i,j The corresponding start time feature, end time feature, Mel frequency cepstral coefficient feature vector, short-time energy feature vector and spectrum envelope feature vector are sequentially arranged to form the corresponding first feature vector a i,j ;

[0142] Step 23: classify each first sound source audio x based on the audio classification model. i,j Perform sound source classification and recognition processing to obtain the corresponding first sound source type b i,j ;

[0143] Among them, the first sound source type b i,j including men, women, children, animals, vehicles and others;

[0144] Here, the audio classification model of the embodiment of the present invention is an end-to-end multi-classification recognition model, and commonly used ones include at least Wav2vec 2.0 model, HuBERT model and VGGish model; the classification types of the audio classification model include men, women, children, animals, vehicles and others;

[0145] Step 24: The total number of first sound sources n i The maximum value of is taken as the corresponding maximum total number of sound sources; and any first total number of sound sources n i The first sound source audio set that matches the maximum total number of sound sources is taken as the corresponding basic sound source audio set; and all other first sound source audio sets except the basic sound source audio set are recorded as the corresponding comparison sound source audio sets;

[0146] Step 25: for all first sound source audios x in the basic sound source audio set i,jPerforming retrieval and clustering processing on the homologous target audio in all the compared sound source audio sets to obtain the corresponding first target complete set;

[0147] The first target full set includes one or more first target audio sets; the number of first target audio sets is the same as the number of first sound source audio sets x in the basic sound source audio set. i,j The same number; each first target audio set includes multiple first sound source audio x i,j ; All first sound source audio x in each first target audio set i,j All of them correspond to the same sound source target. All the first sound source audio x in each first target audio set i,j The corresponding first sound source type b i,j All the same;

[0148] Specifically, step 251 includes: i,j As the corresponding current sound source audio;

[0149] Step 252: The first feature vector a corresponding to the current sound source audio i,j and the first sound source type b i,j As the corresponding current feature vector and current sound source type;

[0150] Step 253, perform a round of traversal on all the comparison sound source audio sets; and in this round of traversal, use the currently traversed comparison sound source audio set as the corresponding current audio set; and i,j The first sound source audio x that matches the current sound source type i,j and when the number of the first preliminary selected audios is not 0, the first feature vector a corresponding to each first preliminary selected audio i,j The similarity with the current feature vector is calculated to obtain the corresponding first similarity; and the largest first similarity is used as the corresponding maximum similarity; and when the maximum similarity exceeds the preset similarity threshold, the first preliminary selected audio corresponding to the maximum similarity is used as a homologous target audio matching the current sound source audio; and at the end of this round of traversal, the current sound source audio and all homologous target audios obtained in this round of traversal are grouped into one category to form a corresponding first target audio set;

[0151] Here, the similarity threshold is a preset percentage threshold;

[0152] Step 254: Check whether the current sound source audio is the last first sound source audio x of the basic sound source audio set. i,j If not, the next first sound source audio x of the basic sound source audio set is i,jExtract it as the new current sound source audio, and return to step 252; if so, all the obtained first target audio sets form the corresponding first target full set;

[0153] Step 26: perform sound source target location estimation based on each first target audio set to obtain the corresponding first target coordinates; and calculate any first sound source audio x in each first target audio set. i,j The corresponding first sound source type b i,j Extract it as the corresponding first target type;

[0154] Specifically, it includes: step 261, performing sound source target positioning estimation based on each first target audio set to obtain corresponding first target coordinates;

[0155] Specifically, it includes: step 261, taking each first target audio set as the corresponding current target audio set; and performing a first sound source audio x i,j The total number of the current audio is counted to obtain the corresponding total number G of the current audio; and the sound source target corresponding to the current target audio set is recorded as the corresponding current sound source target; and the unknown coordinates of the current sound source target are set to the corresponding sound source coordinate variable (x * ,y * ,z * );

[0156] Step 262: any two first sound source audios x in the current target audio set are i,j Denoted as the corresponding microphone audio v p and microphone audio v q Form a corresponding microphone audio pair, 1≤index p≤G, 1≤index q≤G, p≠q; and convert the microphone audio v p 、v q The corresponding microphone coordinates (x, y, z) are recorded as the corresponding microphone coordinates (x p ,y p ,z p ) and microphone coordinates (x q ,y q ,z q ); and the audio v p 、v q The first sampling point time of the corresponding effective signal starting point is recorded as the corresponding microphone arrival time t p and microphone arrival time t q ; and based on the microphone arrival time t p ,t q Calculate the corresponding microphone arrival time difference △t pq =t p -t q ; and the microphone coordinates (xp ,y p ,z p ), microphone coordinates (x q ,y q ,z q ) and the microphone arrival time difference △t pq Form a corresponding microphone pair parameter group;

[0157] Step 263: Calculate the obtained parameters of all microphone pairs and the sound source coordinate variables (x * ,y * ,z * ) construct the corresponding first set of equations;

[0158] The first set of equations is: c×Δt pq =r p -r q ,

[0159] in,

[0160]

[0161] c is the preset sound wave propagation speed parameter, which is set to 343 m / s by default under standard atmospheric pressure.

[0162] r p is the sound source coordinate variable (x * ,y * ,z * ) to the microphone coordinates (x p ,y p ,z p ),

[0163] r q is the sound source coordinate variable (x * ,y * ,z * ) to the microphone coordinates (x q ,y q ,z q ) is the shortest sound wave propagation distance between two objects;

[0164] Step 264, constructing a corresponding first objective function F based on the first set of equations;

[0165] The first objective function F is:

[0166]

[0167] Step 265: calculate the sound source coordinate variable (x* ,y * ,z * ) is solved and the solution is used as the corresponding first target coordinate;

[0168] Here, the embodiment of the present invention can optimize the sound source coordinate variable (x * ,y * ,z * ) is solved by using a genetic algorithm, Newton's method, gradient descent method, particle swarm optimization algorithm, etc.; the optimal solution of the TDOA equation based on any of the above algorithms can be obtained from public technical literature or materials, and the corresponding solution process will not be repeated here;

[0169] Step 262, and convert any first sound source audio x in each first target audio set i,j The corresponding first sound source type b i,j Extract it as the corresponding first target type;

[0170] Step 27, the first target coordinates and the first target type corresponding to each first target audio set form a corresponding first recognition target; and all the obtained first recognition targets form a corresponding first recognition target set.

[0171] Step 3, performing point cloud target recognition processing according to the first environment point cloud to obtain a corresponding second recognition target set;

[0172] The second recognition target set includes one or more second recognition targets; each second recognition target includes second target coordinates, second target type, second target shape, and second target orientation; the second target type includes men, women, children, animals, vehicles, and others;

[0173] Specifically, it includes: step 31, performing target detection and classification recognition processing on the first environment point cloud based on the point cloud target recognition model to obtain a corresponding second detection set;

[0174] The second detection set includes one or more first target detection frames; each first target detection frame includes a first detection frame shape, a first detection frame orientation, a first detection frame center point coordinate, and a first detection frame type; the first detection frame type includes men, women, children, animals, vehicles, and others;

[0175] Here, the point cloud target recognition model of the embodiment of the present invention is an end-to-end point cloud target detection and classification recognition model, and commonly used ones include at least PointRCNN model, VoxelNet model, PointPillars model, PillarNet model, SECOND model and PV-RCNN model;

[0176] Step 32, performing coordinate conversion from the radar coordinate system to the world coordinate system on the center point coordinates of the first detection frame of each first target detection frame to obtain the corresponding world coordinates of the first center point;

[0177] Step 33, a corresponding second recognition target is formed by using the world coordinates of the first center point of each first target detection frame, the first detection frame type, the first detection frame shape and the first detection frame orientation as the corresponding second target coordinates, the second target type, the second target shape and the second target orientation; and a corresponding second recognition target set is formed by all the obtained second recognition targets.

[0178] Step 4, modifying the target type of the second recognition target set based on the first recognition target set;

[0179] Specifically, it includes: performing a round of traversal on all second recognition targets in the second recognition target set; and in this round of traversal, taking the currently traversed second recognition target as the corresponding current target, and taking the second target coordinates and the second target type of the current target as the corresponding current target coordinates and the current target type; and calculating the straight-line distance between each first target coordinate in the first recognition target set and the current target coordinate to obtain the corresponding first distance; and taking the smallest first distance as the corresponding minimum distance; and when the minimum distance does not exceed the preset minimum distance threshold, taking the first target type of the first recognition target corresponding to the minimum distance as the corresponding current comparison target type; and identifying whether the current comparison target type matches the current target type, and if not, correcting the second target type of the current target to the corresponding current comparison target type.

[0180] It should be noted that three types of models will be used in the processing process of the embodiment of the present invention: sound source separation model, audio classification model and point cloud target recognition model; and these three types of models need to complete model training before use, that is: before using the sound source separation model, the sound source separation model is trained based on a preset first data set; before using the audio classification model, the audio classification model is trained based on a preset second data set; before using the point cloud target recognition model, the point cloud target recognition model is trained based on a preset third data set.

[0181] The first, second and third data sets are three preset data sets, which are used to train the sound source separation model, audio classification model and point cloud target recognition model respectively.

[0182] The first data set includes multiple first data records; the first data record includes a first training scene audio and a first label-separated audio set; the first training scene audio is a scene audio containing one or more first sound sources; the target types of the first sound sources include male adults of different ages, female adults of different ages, children of different ages, different kinds of animals, different kinds of vehicles, and other different kinds of sound sources other than humans, animals, and vehicles; the first label-separated audio set includes one or more first label-separated audios; the first label-separated audios of the first label-separated audio set correspond one-to-one to the first sound sources in the first training scene audio; the audio length of each first label-separated audio is the same as the audio length of the corresponding first training scene audio;

[0183] The second data set includes multiple second data records; the second data record includes a first training sound source audio and a first label type; the first training sound source audio is the audio data of a single sound source target; the first label type is the target type of the corresponding single sound source target, including man, woman, child, animal, vehicle and others; the first label type of man indicates that the corresponding single sound source target is a male adult, the first label type of woman indicates that the corresponding single sound source target is a female adult, the first label type of child indicates that the corresponding single sound source target is an underage child, the first label type of animal indicates that the corresponding single sound source target is a type of non-human animal, the first label type of vehicle indicates that the corresponding single sound source target is a type of vehicle, and the first label type of other indicates that the corresponding single sound source target is a type of other sound source type except humans, animals and vehicles;

[0184] The third data set includes multiple third data records; the third data record includes a first training point cloud and a first label detection set; the first label detection set includes one or more first label target detection frames; each first label target detection frame includes a first label detection frame shape, a first label detection frame orientation, a first label detection frame center point coordinate and a first label detection frame type; the first label detection frame types include men, women, children, animals, vehicles and others; if the first label detection frame type is man, it means that the target point cloud in the corresponding first label target detection frame is a male adult, if the first label detection frame type is woman, it means that the target point cloud in the corresponding first label target detection frame is a female adult, if the first label detection frame type is child, it means that the target point cloud in the corresponding first label target detection frame is an underage child, if the first label detection frame type is animal, it means that the target point cloud in the corresponding first label target detection frame is a class of non-human animals, if the first label detection frame type is vehicle, it means that the target point cloud in the corresponding first label target detection frame is a class of vehicles, and if the first label detection frame type is other, it means that the target point cloud in the corresponding first label target detection frame is a class of other target types except humans, animals and vehicles.

[0185] It should also be noted that the specific processing steps of performing model training on the sound source separation model based on the preset first data set before using the sound source separation model in the embodiment of the present invention include:

[0186] Step A1, randomly dividing the first data set into corresponding first training sets and first evaluation sets based on a preset first training evaluation ratio;

[0187] The first training evaluation ratio is a preset ratio, such as 8:2; the first training set and the first evaluation set both include a plurality of first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first training evaluation ratio;

[0188] Step A2, extracting the first first data record in the first training set as the corresponding current training record;

[0189] Step A3, inputting the first training scene audio of the current training record into the sound source separation model to perform sound source audio separation processing to obtain a corresponding first training separation audio set;

[0190] The first training separated audio set includes one or more first training separated audios;

[0191] Step A4, bringing the first training separated audio set and the first label separated audio set of the current training record into a preset first model loss function to calculate and obtain a corresponding first loss value;

[0192] The first model loss function includes an L1 loss function, an L2 loss function, an SDR loss function, an SI-SDR loss function, and a spectrum reconstruction error loss function;

[0193] Step A5, identifying whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identifying whether the current training record is the last first data record in the first training set, if so, turning to step A6, if not, extracting the next first data record in the first training set as a new current training record and returning to step A3; if the first loss value does not satisfy the first loss value range, performing a round of model parameter optimization on the sound source separation model in the direction of minimizing the first model loss function based on a preset first model parameter optimizer, and returning to step A3 at the end of this round of parameter optimization;

[0194] The first loss value range is a preset loss value range; the first model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0195] Step A6, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the currently traversed first data record as the corresponding current evaluation record; and input the first training scene audio of the current evaluation record into the sound source separation model for sound source audio separation processing to obtain the corresponding second training separation audio set; and form a corresponding first prediction label pair from the second training separation audio set and the first label separation audio set of the current evaluation record; and at the end of this round of traversal, bring all the obtained first prediction label pairs into the preset first model evaluation function for calculation to obtain the corresponding first evaluation value;

[0196] Wherein, the first model evaluation function at least includes an MSE error function and an RMSE error function;

[0197] Step A7, identifying whether the first evaluation value satisfies the preset first evaluation value range; if the first evaluation value does not satisfy the first evaluation value range, returning to step A1; if the first evaluation value satisfies the first evaluation value range, solidifying the model parameters of the sound source separation model and confirming that the current model training of the sound source separation model is completed.

[0198] Here, the first evaluation value range is a preset evaluation value range.

[0199] It should also be noted that the specific processing steps of performing model training on the audio classification model based on the preset second data set before using the audio classification model in the embodiment of the present invention include:

[0200] Step B1, randomly dividing the second data set into corresponding second training sets and second evaluation sets based on a preset second training evaluation ratio;

[0201] The second training evaluation ratio is a preset ratio, such as 8:2; the second training set and the second evaluation set both include a plurality of second data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second training evaluation ratio;

[0202] Step B2, extracting the first second data record in the second training set as the corresponding current training record;

[0203] Step B3, inputting the first training sound source audio of the current training record into the audio classification model for sound source classification and recognition processing to obtain the corresponding first training sound source type;

[0204] The first training sound source types include men, women, children, animals, vehicles and others;

[0205] Step B4, bringing the first training sound source type and the first label type of the current training record into a preset second model loss function to calculate and obtain a corresponding second loss value;

[0206] Wherein, the second model loss function includes a multi-classification cross entropy loss function;

[0207] Step B5, identifying whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, identifying whether the current training record is the last second data record in the second training set, if so, turning to step B6, if not, extracting the next second data record in the second training set as a new current training record and returning to step B3; if the second loss value does not satisfy the second loss value range, performing a round of model parameter optimization on the audio classification model in the direction of minimizing the second model loss function based on a preset second model parameter optimizer, and returning to step B3 at the end of this round of parameter optimization;

[0208] The second loss value range is a preset loss value range; the second model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0209] Step B6, perform a round of traversal on all the second data records of the second evaluation set; and in this round of traversal, use the currently traversed second data record as the corresponding current evaluation record; and input the first training sound source audio of the current evaluation record into the audio classification model for sound source classification and recognition processing to obtain the corresponding second training sound source type; and form a corresponding second predicted label pair by the second training sound source type and the first label type of the current evaluation record; and at the end of this round of traversal, calculate the precision and recall based on all the obtained second predicted label pairs to obtain the corresponding first precision and first recall; and calculate the F1 score based on the first precision and the first recall to obtain the corresponding first F1 score; and use the first F1 score as the corresponding second evaluation value;

[0210] Step B7, identify whether the second evaluation value meets the preset second evaluation value range; if the second evaluation value does not meet the second evaluation value range, return to step B1; if the second evaluation value meets the second evaluation value range, solidify the model parameters of the audio classification model and confirm that the current model training of the audio classification model is completed.

[0211] Here, the second evaluation value range is a preset evaluation value range.

[0212] It should also be noted that the specific processing steps of performing model training on the point cloud target recognition model based on the preset third data set before using the point cloud target recognition model in the embodiment of the present invention include:

[0213] Step C-1, randomly dividing the third data set into corresponding third training sets and third evaluation sets based on a preset third training evaluation ratio;

[0214] The third training evaluation ratio is a preset ratio, such as 8:2; the third training set and the third evaluation set both include a plurality of third data records; the ratio of the total number of records in the third training set to the total number of records in the third evaluation set satisfies the third training evaluation ratio;

[0215] Step C-2, extracting the first third data record in the third training set as the corresponding current training record;

[0216] Step C-3, inputting the first training point cloud of the current training record into the point cloud target recognition model for target detection and classification recognition processing to obtain a corresponding first training detection set;

[0217] The first training detection set includes one or more first training target detection frames; each first training target detection frame includes a first training detection frame shape, a first training detection frame orientation, a first training detection frame center point coordinate, and a first training detection frame type; the first training detection frame types include men, women, children, animals, vehicles, and others;

[0218] Step C-4, taking each first label target detection frame of the first label detection set of the current training record as the corresponding current label frame; and taking the first training target detection frame in the first training detection set that is closest to the current label frame as the corresponding current training frame; and constructing a vector length of 6 full zero vectors for the current label frame and the current training frame as the corresponding first label type vector and first training type vector, respectively, and establishing a one-to-one correspondence between the 6 vector data of the first label type vector and the first training type vector and the 6 classification types; and forming a corresponding first label detection frame shape, first label detection frame orientation, and first label detection frame center point coordinates of the current label frame The first training geometric vector of the current training frame is composed of the first training detection frame shape, the first training detection frame orientation, and the first training detection frame center point coordinates to form a corresponding first label geometric vector; and the vector data corresponding to the first label detection frame type of the current label frame in the first label type vector is reset to 1; and the vector data corresponding to the first training detection frame type of the current training frame in the first training type vector is reset to 1; and the first training geometric vector corresponding to the current label frame and the first label geometric vector form a corresponding first geometric vector pair; and the first training type vector corresponding to the current label frame and the first label type vector form a corresponding first type vector pair;

[0219] Among them, the six classification types include men, women, children, animals, vehicles and others;

[0220] Step C-5, bringing all first geometric vector pairs corresponding to the current training record into the preset third model loss function to calculate the corresponding third loss value; bringing all first type vector pairs corresponding to the current training record into the preset fourth model loss function to calculate the corresponding fourth loss value; and performing weighted sum calculation on the third and fourth loss values ​​to obtain the corresponding fifth loss value;

[0221] The third model loss function includes L1 loss function and L2 loss function;

[0222] The fourth model loss function includes a multi-classification cross entropy loss function;

[0223] Step C-6, identifying whether the fifth loss value satisfies the preset third loss value range; if the fifth loss value satisfies the third loss value range, identifying whether the current training record is the last third data record in the third training set, if so, turning to step C-7, if not, extracting the next third data record in the third training set as the new current training record and returning to step C-3; if the fifth loss value does not satisfy the third loss value range, then based on the preset third model parameter optimizer, a round of model parameter optimization is performed on the point cloud target recognition model in the direction of minimizing the third and fourth model loss functions, and returning to step C-3 at the end of this round of parameter optimization;

[0224] The third loss value range is a preset loss value range; the third model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0225] Step C-7, inputting the first training point cloud of each third data record of the third evaluation set into the point cloud target recognition model for target detection and classification recognition processing to obtain a corresponding second training detection set;

[0226] The second training detection set includes one or more second training target detection frames; each second training target detection frame includes a second training detection frame shape, a second training detection frame orientation, a second training detection frame center point coordinate, and a second training detection frame type; the second training detection frame types include men, women, children, animals, vehicles, and others;

[0227] Step C-8, taking the first label detection set of each third data record of the third evaluation set as the corresponding current label detection set; and taking the second training detection set corresponding to the current label detection set as the corresponding current training detection set; and taking each first label target detection frame in the current label detection set as the corresponding current label frame; and forming a corresponding second training geometric vector from the first label detection frame shape, the first label detection frame orientation, and the first label detection frame center point coordinates of the current label frame, and forming a corresponding second label geometric vector from the second training detection frame shape, the second training detection frame orientation, and the second training detection frame center point coordinates of the current training frame; and forming a corresponding second geometric vector pair from the second training geometric vector corresponding to the current label frame and the second label geometric vector;

[0228] Step C-9, taking the first label detection set of each third data record of the third evaluation set as the corresponding current label detection set; and taking the second training detection set corresponding to the current label detection set as the corresponding current training detection set; and taking each first label target detection frame in the current label detection set as the corresponding current label frame; and taking the second training target detection frame in the current training detection set that is closest to the current label frame as the corresponding current training frame; and taking the first label detection frame type of the current label frame as the corresponding current label type, and taking the second training detection frame type of the current training frame as the corresponding current training type; and constructing a full zero vector with a vector length of 6 for the current label frame and the current training frame respectively as the corresponding second label type vector and second training type vector; and establishing a one-to-one correspondence between the 6 vector data of the second label type vector and the second training type vector and the 6 classification types; and resetting the vector data corresponding to the current label type in the second label type vector to 1, and resetting the vector data corresponding to the previous training type in the second training type vector to 1; and forming a corresponding second type vector pair by the second training type vector and the second label type vector corresponding to the current label frame;

[0229] Step C-10, bringing all the obtained second geometric vector pairs into the preset second model evaluation function to calculate and obtain the corresponding third evaluation value; and calculating the precision and recall rate according to all the obtained second type vector pairs to obtain the corresponding second precision and second recall rate, and calculating the F1 score based on the second precision and the second recall rate to obtain the corresponding second F1 score, and using the second F1 score as the corresponding fourth evaluation value;

[0230] Wherein, the second model evaluation function includes at least an MSE error function and an RMSE error function;

[0231] Step C-11, and identify the third and fourth evaluation values; if the third evaluation value does not meet the preset third evaluation value range or the fourth evaluation value does not meet the preset fourth evaluation value range, return to step C-1; if the third evaluation value meets the preset third evaluation value range and the fourth evaluation value meets the preset fourth evaluation value range, the model parameters of the point cloud target recognition model are solidified once and confirm that the current model training of the point cloud target recognition model is completed.

[0232] Here, the third and fourth evaluation value ranges are two preset evaluation value ranges.

[0233] Figure 2 The module structure diagram of a device for target recognition based on audio and laser radar point cloud provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device for target recognition based on audio and laser radar point cloud includes: a data acquisition module 201, an audio target recognition processing module 202, a point cloud target recognition processing module 203 and a target recognition result correction module 204.

[0234] The data acquisition module 201 is used to collect audio of the current environment based on the first directional microphone array to obtain a corresponding first array audio set; and during the audio collection process, perform a radar scan on the current environment based on the first laser radar to obtain a corresponding first environment point cloud.

[0235] The audio target recognition processing module 202 is used to perform sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set; the first recognition target set includes one or more first recognition targets; each first recognition target includes first target coordinates and a first target type; the first target types include men, women, children, animals, vehicles and others.

[0236] The point cloud target recognition processing module 203 is used to perform point cloud target recognition processing according to the first environment point cloud to obtain a corresponding second recognition target set; the second recognition target set includes one or more second recognition targets; each second recognition target includes second target coordinates, a second target type, a second target shape and a second target orientation; the second target types include men, women, children, animals, vehicles and others.

[0237] The target recognition result correction module 204 is used to correct the target type of the second recognition target set based on the first recognition target set.

[0238] An embodiment of the present invention provides a device for target recognition based on audio and lidar point cloud, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0239] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the data acquisition module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0240] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0241] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0242] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0243] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0244] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0245] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0246] An embodiment of the present invention further provides a chip for executing instructions, wherein the chip is used to execute the processing steps described in the aforementioned method embodiment.

[0247] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for target recognition based on audio and laser radar point cloud. It can be seen from the above invention content that the embodiments of the present invention provide a sensor combination of a directional microphone array + laser radar, and the directional microphone array of the combination performs audio collection on the current environment to obtain a corresponding microphone audio set, and the laser radar of the combination performs radar scanning on the current environment to obtain a corresponding environmental point cloud; and performs sound source target recognition processing based on the microphone audio set to obtain a corresponding first recognition target set, and performs point cloud target recognition processing based on the environmental point cloud to obtain a corresponding second recognition target set; and uses the first recognition target set to correct the target type of the second recognition target set. The sensor combination of a directional microphone array + laser radar provided in the embodiments of the present invention can be used as a supplementary solution to the traditional camera + laser radar combination. The sensor combination of the embodiments of the present invention can effectively correct the point cloud target recognition results based on the audio target recognition results in an environment with bad weather or poor visibility.

[0248] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0249] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0250] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for target recognition based on audio and laser radar point cloud, characterized in that: The method comprises: Based on the first directional microphone array, audio of the current environment is collected to obtain a corresponding first array audio set; and during the audio collection process, radar scanning of the current environment is performed based on the first laser radar to obtain a corresponding first environment point cloud; Performing sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set; the first recognition target set includes one or more first recognition targets; each of the first recognition targets includes first target coordinates and a first target type; the first target types include men, women, children, animals, vehicles and others; Performing point cloud target recognition processing according to the first environmental point cloud to obtain a corresponding second recognition target set; the second recognition target set includes one or more second recognition targets; each of the second recognition targets includes second target coordinates, second target type, second target shape, and second target orientation; the second target types include men, women, children, animals, vehicles, and others; The target type of the second recognition target set is corrected based on the first recognition target set.

2. The method for target recognition based on audio and laser radar point cloud according to claim 1, characterized in that: The first directional microphone array includes a plurality of first directional microphones M i , 1≤index i≤N ini , N ini is the total number of directional microphones in the first directional microphone array; each of the first directional microphones M i Corresponding to a microphone coordinate (x, y, z), each of the microphone coordinates (x, y, z) is a world coordinate system coordinate; The first array audio set includes a plurality of first microphone audio X i ; The first microphone audio X i With the first directional microphone M i One-to-one correspondence; each of the first microphone audio X i is the first directional microphone M corresponding to i The ambient audio obtained by collecting audio from the current environment; all the first directional microphones M i When the audio of the current environment is collected, it is collected synchronously, that is, all the first microphone audio X i The collection start time, collection end time and total collection duration are the same.

3. The method for target recognition based on audio and laser radar point cloud according to claim 2, characterized in that: The performing sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set specifically includes: Based on the sound source separation model, the first microphone audio X i Perform sound source audio separation processing to obtain a corresponding first sound source audio set; and count the total number of sound source audios in each of the first sound source audio sets to obtain a corresponding first sound source total number n i ; The sound source separation model includes at least a TasNet model, a Conv-TasNet model, a Multi-StreamConv-TasNet model, a Wave-U-Net model, a MulCat DPRNN model and a DPTNet model; The first sound source audio set includes one or more first sound source audio x i,j , 1≤indexj≤n i Each of the first sound source audio x i,j The length of time corresponding to the first microphone audio X i The length of time is consistent; For each of the first sound source audio x i,j Perform sound source feature extraction to obtain the corresponding first feature vector a i,j ; The first feature vector a i,j Including start time feature, end time feature, Mel frequency cepstral coefficient feature vector, short time energy feature vector and spectrum envelope feature vector; Based on the audio classification model, each of the first sound source audio x i,j Perform sound source classification and recognition processing to obtain the corresponding first sound source type b i,j ; The audio classification model includes at least Wav2vec 2.0 model, HuBERT model and VGGish model; The first sound source type b i,j including men, women, children, animals, vehicles and others; The total number of the first sound sources n i The maximum value of the corresponding maximum total number of sound sources; and any of the first total number of sound sources n i The first sound source audio set that matches the maximum total number of sound sources is used as the corresponding basic sound source audio set; and all other first sound source audio sets except the basic sound source audio set are recorded as corresponding comparison sound source audio sets; For all the first sound source audio x in the basic sound source audio set i,j The homologous target audio in all the compared sound source audio sets is retrieved and clustered to obtain a corresponding first target set; the first target set includes one or more first target audio sets; the number of the first target audio sets is the same as the number of the first sound source audio x in the basic sound source audio set. i,j The same number; each of the first target audio sets includes a plurality of the first sound source audio x i,j ; All the first sound source audio x in each of the first target audio sets i,j All of the first sound source audio x in each of the first target audio sets correspond to the same sound source target. i,j The corresponding first sound source type b i,j All the same; Based on each of the first target audio sets, the sound source target positioning estimation is performed to obtain the corresponding first target coordinates; and any one of the first sound source audio x in each of the first target audio sets is i,j The corresponding first sound source type b i,j extracting it as the corresponding first target type; The first target coordinates and the first target type corresponding to each of the first target audio sets form a corresponding first recognition target; and all the obtained first recognition targets form a corresponding first recognition target set.

4. The method for target recognition based on audio and laser radar point cloud according to claim 3, characterized in that: The first sound source audio x i,j Perform sound source feature extraction to obtain the corresponding first feature vector a i,j , specifically including: The first sound source audio x is sampled at a preset first audio sampling frequency. i,j Performing audio signal sampling processing to obtain a corresponding first signal sampling sequence; the first signal sampling sequence is formed by sequentially sorting a plurality of first signal sampling points; each of the first signal sampling points includes a first sampling point time and a first sampling point amplitude; The first and last first signal sampling points of the first signal sampling sequence are recorded as the corresponding starting sampling point and ending sampling point; and the only first signal sampling point in the first signal sampling sequence whose amplitudes of all the first sampling points between the starting sampling point and the first signal sampling point are 0 is recorded as the corresponding valid signal starting point, and the time interval between the valid signal starting point and the starting sampling point is calculated to obtain the corresponding starting time interval △t s ; and the only first signal sampling point in the first signal sampling sequence whose amplitudes of all the first sampling points between the end sampling point and the end sampling point are 0 is recorded as the corresponding valid signal end point, and the time interval between the end sampling point and the valid signal end point is calculated to obtain the corresponding end time interval △t e ; and count the total number of the first signal sampling points of the first signal sampling sequence and use the statistical result as the corresponding first sequence length L; and based on the starting time interval △t s , the end time interval Δt e The starting time feature and the ending time feature corresponding to the first sequence length L are set, and the starting time feature = Δt s / L, end time characteristic = 1-△t e / L; The first signal sampling sequence is subjected to signal framing processing according to a preset audio framing rule to obtain a corresponding first signal frame sequence; the first signal frame sequence is formed by sequentially sorting one or more first signal frames; each of the first signal frames is formed by sequentially sorting a plurality of first signal sampling points; the number of the first signal sampling points of all the first signal frames is the same; Identify the Mel-frequency cepstral coefficient feature vectors of each of the first signal frames to obtain a corresponding first frame coefficient feature vector; and sequentially sort all the obtained first frame coefficient feature vectors to form the corresponding Mel-frequency cepstral coefficient feature vector; Calculate the signal energy of each of the first signal frames and use the calculation result as the corresponding first frame signal energy; and sort all the obtained first frame signal energies in order to form the corresponding short-time energy feature vector; Identifying the spectrum envelope feature vectors of each of the first signal frames to obtain a corresponding first frame envelope feature vector; and sequentially sorting all the obtained first frame envelope feature vectors to form a corresponding spectrum envelope feature vector; The first sound source audio x i,j The corresponding start time feature, the end time feature, the Mel frequency cepstral coefficient feature vector, the short-time energy feature vector and the spectrum envelope feature vector are sequentially arranged to form the corresponding first feature vector a i,j .

5. The method for target recognition based on audio and laser radar point cloud according to claim 3, characterized in that: The method for all the first sound source audio x in the basic sound source audio set i,j Performing a search and clustering process on the homologous target audios in all the compared sound source audio sets to obtain a corresponding first target full set specifically includes: Step 51: The first audio source x of the basic audio source set is i,j As the corresponding current sound source audio; Step 52: convert the first feature vector a corresponding to the current sound source audio i,j and the first sound source type b i,j As the corresponding current feature vector and current sound source type; Step 53, perform a round of traversal on all the compared sound source audio sets; and in this round of traversal, use the currently traversed compared sound source audio set as the corresponding current audio set; and add the first sound source type b in the current audio set. i,j The first sound source audio x that matches the current sound source type i,j and when the number of the first preliminary selected audios is not 0, the first feature vector a corresponding to each of the first preliminary selected audios is i,j The similarity with the current feature vector is calculated to obtain a corresponding first similarity; and the largest of the first similarities is used as the corresponding maximum similarity; and when the maximum similarity exceeds a preset similarity threshold, the first preliminary selected audio corresponding to the maximum similarity is used as a homologous target audio matching the current sound source audio; and at the end of this round of traversal, the current sound source audio and all the homologous target audios obtained in this round of traversal are grouped into one category to form a corresponding first target audio set; Step 54: Check whether the current sound source audio is the last of the first sound source audio x in the basic sound source audio set. i,j If not, the next first sound source audio x of the basic sound source audio set is i,j Extract it as the new current sound source audio, and return to step 52; if so, all the first target audio sets obtained form the corresponding first target full set.

6. The method for target recognition based on audio and laser radar point cloud according to claim 4, characterized in that: The performing sound source target positioning estimation based on each of the first target audio sets to obtain the corresponding first target coordinates specifically includes: Step 61: taking each of the first target audio sets as the corresponding current target audio set; and performing a calculation on the first sound source audio x of the current target audio set. i,j The total number of the current audio is counted to obtain the corresponding total number G of the current audio; and the sound source target corresponding to the current target audio set is recorded as the corresponding current sound source target; and the unknown coordinates of the current sound source target are set to the corresponding sound source coordinate variable (x * ,y * ,z * ); Step 62: any two of the first sound source audios x in the current target audio set are combined into i,j Denoted as the corresponding microphone audio v p and microphone audio v q Form a corresponding microphone audio pair, 1≤index p≤G, 1≤index q≤G, p≠q; and convert the microphone audio v p 、v q The corresponding microphone coordinates (x, y, z) are recorded as the corresponding microphone coordinates (x p ,y p ,z p ) and microphone coordinates (x q ,y q ,z q ); and the audio v p 、v q The first sampling point time corresponding to the starting point of the effective signal is recorded as the corresponding microphone arrival time t p and microphone arrival time t q ; and based on the microphone arrival time t p ,t q Calculate the corresponding microphone arrival time difference △t pq =t p -t q ; and the microphone coordinates (x p ,y p ,z p ), the microphone coordinates (x q ,y q ,z q ) and the microphone arrival time difference △t pq Form a corresponding microphone pair parameter group; Step 63: According to the TDOA algorithm, all the microphone pair parameter groups and the sound source coordinate variables (x * ,y * ,z * ) construct the corresponding first system of equations; The first set of equations is: c×Δt pq =r p -r q , in, c is the preset sound wave propagation speed parameter, r p is the sound source coordinate variable (x * ,y * ,z * ) to the microphone coordinates (x p ,y p ,z p ), r q is the sound source coordinate variable (x * ,y * ,z * ) to the microphone coordinates (x q ,y q ,z q ) is the shortest sound wave propagation distance between two objects; Step 64, constructing a corresponding first objective function F based on the first set of equations; The first objective function F is: Step 65: the sound source coordinate variable (x * ,y * ,z * ) is solved and the solution is used as the corresponding first target coordinate.

7. The method for target recognition based on audio and laser radar point cloud according to claim 1, characterized in that: The step of performing point cloud target recognition processing according to the first environment point cloud to obtain a corresponding second recognition target set specifically includes: The first environment point cloud is subjected to target detection and classification recognition processing based on a point cloud target recognition model to obtain a corresponding second detection set; the point cloud target recognition model includes at least a PointRCNN model, a VoxelNet model, a PointPillars model, a PillarNet model, a SECOND model, and a PV-RCNN model; the second detection set includes one or more first target detection frames; each of the first target detection frames includes a first detection frame shape, a first detection frame orientation, a first detection frame center point coordinate, and a first detection frame type; the first detection frame types include men, women, children, animals, vehicles, and others; Performing coordinate conversion on the center point coordinates of the first detection frame of each first target detection frame from the radar coordinate system to the world coordinate system to obtain the corresponding first center point world coordinates; A corresponding second recognition target is formed by the world coordinates of the first center point of each first target detection frame, the first detection frame type, the first detection frame shape and the first detection frame orientation as the corresponding second target coordinates, the second target type, the second target shape and the second target orientation; and a corresponding second recognition target set is formed by all the obtained second recognition targets.

8. The method for target recognition based on audio and laser radar point cloud according to claim 1, characterized in that: The performing target type correction on the second identified target set based on the first identified target set specifically includes: A round of traversal is performed on all the second identified targets in the second identified target set; and in this round of traversal, the second identified target currently traversed is taken as the corresponding current target, and the second target coordinates and the second target type of the current target are taken as the corresponding current target coordinates and the current target type; and the straight-line distances between each of the first target coordinates in the first identified target set and the current target coordinates are calculated to obtain the corresponding first distance; and the smallest first distance is taken as the corresponding minimum distance; and when the minimum distance does not exceed the preset minimum distance threshold, the first target type of the first identified target corresponding to the minimum distance is taken as the corresponding current comparison target type; and whether the current comparison target type matches the current target type is identified, and if not, the second target type of the current target is corrected to the corresponding current comparison target type.

9. The method for target recognition based on audio and laser radar point cloud according to claim 3, characterized in that: The method further comprises: Before using the sound source separation model, the sound source separation model is trained based on a preset first data set; wherein the first data set includes multiple first data records; the first data record includes a first training scene audio and a first label separation audio set; the first training scene audio is a scene audio containing one or more first sound sources; the target types of the first sound source include male adults of different ages, female adults of different ages, children of different ages, different kinds of animals, different kinds of vehicles, and other different kinds of sound sources except humans, animals, and vehicles; the first label separation audio set includes one or more first label separation audios; the first label separation audio of the first label separation audio set corresponds one-to-one to the first sound source in the first training scene audio; the audio length of each first label separation audio is the same as the audio length of the corresponding first training scene audio; Before using the audio classification model, the audio classification model is trained based on a preset second data set; wherein the second data set includes multiple second data records; the second data record includes a first training sound source audio and a first label type; the first training sound source audio is the audio data of a single sound source target; the first label type is the target type of the corresponding single sound source target, including man, woman, child, animal, vehicle and others; the first label type of man indicates that the corresponding single sound source target is a male adult, the first label type of woman indicates that the corresponding single sound source target is a female adult, the first label type of child indicates that the corresponding single sound source target is an underage child, the first label type of animal indicates that the corresponding single sound source target is a type of non-human animal, the first label type of vehicle indicates that the corresponding single sound source target is a type of vehicle, and the first label type of other indicates that the corresponding single sound source target is a type of sound source other than humans, animals and vehicles.

10. The method for target recognition based on audio and laser radar point cloud according to claim 7, characterized in that: The method further comprises: Before using the point cloud target recognition model, the point cloud target recognition model is trained based on a preset third data set; wherein the third data set includes multiple third data records; the third data record includes a first training point cloud and a first label detection set; the first label detection set includes one or more first label target detection frames; each of the first label target detection frames includes a first label detection frame shape, a first label detection frame orientation, a first label detection frame center point coordinates and a first label detection frame type; the first label detection frame types include men, women, children, animals, vehicles and others; the first label detection frame type is a man, indicating that the corresponding target point cloud in the first label target detection frame is a If the first label detection frame type is woman, the target point cloud in the first label target detection frame is a female adult; if the first label detection frame type is child, the target point cloud in the first label target detection frame is an underage child; if the first label detection frame type is animal, the target point cloud in the first label target detection frame is a type of non-human animal; if the first label detection frame type is vehicle, the target point cloud in the first label target detection frame is a type of vehicle; if the first label detection frame type is other, the target point cloud in the first label target detection frame is a type of target other than humans, animals, and vehicles.

11. The method for target recognition based on audio and laser radar point cloud according to claim 9, characterized in that: The performing model training on the sound source separation model based on a preset first data set before using the sound source separation model specifically includes: Step 111, randomly dividing the first data set into corresponding first training sets and first evaluation sets based on a preset first training evaluation ratio; Wherein, both the first training set and the first evaluation set include a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first training evaluation ratio; Step 112, extracting the first first data record in the first training set as the corresponding current training record; Step 113, inputting the first training scene audio of the current training record into the sound source separation model to perform sound source audio separation processing to obtain a corresponding first training separation audio set; Wherein, the first training separated audio set includes one or more first training separated audios; Step 114, bringing the first training separated audio set and the first label separated audio set of the current training record into a preset first model loss function to calculate and obtain a corresponding first loss value; Wherein, the first model loss function includes an L1 loss function, an L2 loss function, an SDR loss function, an SI-SDR loss function and a spectrum reconstruction error loss function; Step 115, identifying whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identifying whether the current training record is the last first data record in the first training set, if so, turning to step 116, if not, extracting the next first data record in the first training set as the new current training record and returning to step 113; if the first loss value does not satisfy the first loss value range, performing a round of model parameter optimization on the sound source separation model in the direction of minimizing the first model loss function based on a preset first model parameter optimizer, and returning to step 113 at the end of this round of parameter optimization; Wherein, the first model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 116, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the first training scene audio of the current evaluation record into the sound source separation model for sound source audio separation processing to obtain the corresponding second training separation audio set; and form a corresponding first prediction label pair by the second training separation audio set and the first label separation audio set of the current evaluation record; and at the end of this round of traversal, bring all the first prediction label pairs obtained into the preset first model evaluation function for calculation to obtain the corresponding first evaluation value; Wherein, the first model evaluation function at least includes an MSE error function and an RMSE error function; Step 117, identifying whether the first evaluation value satisfies a preset first evaluation value range; if the first evaluation value does not satisfy the first evaluation value range, returning to step 111; if the first evaluation value satisfies the first evaluation value range, solidifying the model parameters of the sound source separation model and confirming that the current model training of the sound source separation model is completed.

12. The method for target recognition based on audio and laser radar point cloud according to claim 9, characterized in that: The performing model training on the audio classification model based on a preset second data set before using the audio classification model specifically includes: Step 121, randomly dividing the second data set into corresponding second training sets and second evaluation sets based on a preset second training evaluation ratio; Wherein, both the second training set and the second evaluation set include a plurality of second data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second training evaluation ratio; Step 122, extracting the first second data record in the second training set as the corresponding current training record; Step 123, inputting the first training sound source audio of the current training record into the audio classification model to perform sound source classification and recognition processing to obtain a corresponding first training sound source type; Wherein, the first training sound source types include men, women, children, animals, vehicles and others; Step 124, bringing the first training sound source type and the first label type of the current training record into a preset second model loss function to calculate and obtain a corresponding second loss value; Wherein, the second model loss function includes a multi-classification cross entropy loss function; Step 125, identifying whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, identifying whether the current training record is the last second data record in the second training set, if so, turning to step 126, if not, extracting the next second data record in the second training set as the new current training record and returning to step 123; if the second loss value does not satisfy the second loss value range, performing a round of model parameter optimization on the audio classification model in the direction of minimizing the second model loss function based on a preset second model parameter optimizer, and returning to step 123 at the end of this round of parameter optimization; Wherein, the second model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 126, perform a round of traversal on all the second data records of the second evaluation set; and in this round of traversal, use the second data record currently traversed as the corresponding current evaluation record; and input the first training sound source audio of the current evaluation record into the audio classification model for sound source classification and recognition processing to obtain the corresponding second training sound source type; and form a corresponding second predicted label pair by the second training sound source type and the first label type of the current evaluation record; and at the end of this round of traversal, calculate the precision and recall based on all the obtained second predicted label pairs to obtain the corresponding first precision and first recall; and calculate the F1 score based on the first precision and the first recall to obtain the corresponding first F1 score; and use the first F1 score as the corresponding second evaluation value; Step 127, identifying whether the second evaluation value satisfies a preset second evaluation value range; if the second evaluation value does not satisfy the second evaluation value range, returning to step 121; if the second evaluation value satisfies the second evaluation value range, solidifying the model parameters of the audio classification model and confirming that the current model training of the audio classification model is completed.

13. The method for target recognition based on audio and laser radar point cloud according to claim 10, characterized in that: The performing model training on the point cloud target recognition model based on a preset third data set before using the point cloud target recognition model specifically includes: Step 131, randomly dividing the third data set into corresponding third training sets and third evaluation sets based on a preset third training evaluation ratio; Wherein, the third training set and the third evaluation set both include a plurality of the third data records; the ratio of the total number of records in the third training set to the total number of records in the third evaluation set satisfies the third training evaluation ratio; Step 132, extracting the first third data record in the third training set as the corresponding current training record; Step 133, inputting the first training point cloud of the current training record into the point cloud target recognition model to perform target detection and classification recognition processing to obtain a corresponding first training detection set; The first training detection set includes one or more first training target detection frames; each of the first training target detection frames includes a first training detection frame shape, a first training detection frame orientation, a first training detection frame center point coordinate, and a first training detection frame type; the first training detection frame types include men, women, children, animals, vehicles, and others; Step 134, taking each of the first label target detection frames of the first label detection set of the current training record as the corresponding current label frame; and taking the first training target detection frame in the first training detection set that is closest to the current label frame as the corresponding current training frame; and constructing a full zero vector with a vector length of 6 for the current label frame and the current training frame as the corresponding first label type vector and first training type vector, respectively, and establishing a one-to-one correspondence between the 6 vector data of the first label type vector and the first training type vector and the 6 classification types; and forming a corresponding first label detection frame shape, the first label detection frame orientation, and the first label detection frame center point coordinates of the current label frame. a training geometric vector, and a corresponding first label geometric vector is formed by the shape of the first training detection frame, the orientation of the first training detection frame, and the coordinates of the center point of the first training detection frame of the current training frame; and the vector data corresponding to the first label detection frame type of the current label frame in the first label type vector is reset to 1; and the vector data corresponding to the first training detection frame type of the current training frame in the first training type vector is reset to 1; and the first training geometric vector corresponding to the current label frame and the first label geometric vector form a corresponding first geometric vector pair; and the first training type vector corresponding to the current label frame and the first label type vector form a corresponding first type vector pair; The six classification types include men, women, children, animals, vehicles and others; Step 135, bringing all the first geometric vector pairs corresponding to the current training record into a preset third model loss function to calculate a corresponding third loss value; bringing all the first type vector pairs corresponding to the current training record into a preset fourth model loss function to calculate a corresponding fourth loss value; and performing weighted sum calculation on the third and fourth loss values ​​to obtain a corresponding fifth loss value; The third model loss function includes an L1 loss function and an L2 loss function; The fourth model loss function includes a multi-classification cross entropy loss function; Step 136, identifying whether the fifth loss value satisfies a preset third loss value range; if the fifth loss value satisfies the third loss value range, identifying whether the current training record is the last third data record in the third training set, if so, turning to step 137, if not, extracting the next third data record in the third training set as the new current training record and returning to step 133; if the fifth loss value does not satisfy the third loss value range, performing a round of model parameter optimization on the point cloud target recognition model in the direction of minimizing the third and fourth model loss functions based on a preset third model parameter optimizer, and returning to step 133 at the end of this round of parameter optimization; Wherein, the third model parameter optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 137, inputting the first training point cloud of each of the third data records in the third evaluation set into the point cloud target recognition model for target detection and classification recognition processing to obtain a corresponding second training detection set; The second training detection set includes one or more second training target detection frames; each of the second training target detection frames includes a second training detection frame shape, a second training detection frame orientation, a second training detection frame center point coordinate, and a second training detection frame type; the second training detection frame types include men, women, children, animals, vehicles, and others; Step 138, taking the first label detection set of each of the third data records in the third evaluation set as the corresponding current label detection set; and taking the second training detection set corresponding to the current label detection set as the corresponding current training detection set; and taking each of the first label target detection frames in the current label detection set as the corresponding current label frame; and forming a corresponding second training geometric vector by the shape of the first label detection frame, the orientation of the first label detection frame, and the coordinates of the center point of the first label detection frame of the current label frame, and forming a corresponding second label geometric vector by the shape of the second training detection frame, the orientation of the second training detection frame, and the coordinates of the center point of the second training detection frame of the current training frame; and forming a corresponding second geometric vector pair by the second training geometric vector corresponding to the current label frame and the second label geometric vector; Step 139, taking the first label detection set of each of the third data records of the third evaluation set as the corresponding current label detection set; taking the second training detection set corresponding to the current label detection set as the corresponding current training detection set; taking each of the first label target detection frames in the current label detection set as the corresponding current label frame; taking the second training target detection frame in the current training detection set that is closest to the current label frame as the corresponding current training frame; taking the first label detection frame type of the current label frame as the corresponding current label type, and taking the second training detection frame type of the current training frame as the corresponding current training type; and constructing a full zero vector with a vector length of 6 for the current label frame and the current training frame respectively as the corresponding second label type vector and second training type vector; and establishing a one-to-one correspondence between the 6 vector data of the second label type vector and the second training type vector and the 6 classification types; and resetting the vector data corresponding to the current label type in the second label type vector to 1, and resetting the vector data corresponding to the previous training type in the second training type vector to 1; and forming a corresponding second type vector pair by the second training type vector corresponding to the current label frame and the second label type vector; Step 140, bringing all the obtained second geometric vector pairs into a preset second model evaluation function to calculate and obtain a corresponding third evaluation value; and performing precision and recall calculations on all the obtained second type vector pairs to obtain a corresponding second precision and second recall, and performing F1 score calculations based on the second precision and the second recall to obtain a corresponding second F1 score, and using the second F1 score as the corresponding fourth evaluation value; Wherein, the second model evaluation function at least includes an MSE error function and an RMSE error function; Step 141, and identify the third and fourth evaluation values; if the third evaluation value does not meet the preset third evaluation value range or the fourth evaluation value does not meet the preset fourth evaluation value range, return to step 131; if the third evaluation value meets the preset third evaluation value range and the fourth evaluation value meets the preset fourth evaluation value range, the model parameters of the point cloud target recognition model are solidified once and it is confirmed that the current model training of the point cloud target recognition model is completed.

14. A device for executing the method for target recognition based on audio and laser radar point cloud according to any one of claims 1 to 13, characterized in that: The device comprises: a data acquisition module, an audio target recognition processing module, a point cloud target recognition processing module and a target recognition result correction module; The data acquisition module is used to collect audio of the current environment based on the first directional microphone array to obtain a corresponding first array audio set; and to perform radar scanning on the current environment based on the first laser radar to obtain a corresponding first environment point cloud during the audio acquisition process; The audio target recognition processing module is used to perform sound source target recognition processing according to the first array audio set to obtain a corresponding first recognition target set; the first recognition target set includes one or more first recognition targets; each of the first recognition targets includes first target coordinates and a first target type; the first target type includes men, women, children, animals, vehicles and others; The point cloud target recognition processing module is used to perform point cloud target recognition processing according to the first environmental point cloud to obtain a corresponding second recognition target set; the second recognition target set includes one or more second recognition targets; each second recognition target includes second target coordinates, second target type, second target shape and second target orientation; the second target type includes men, women, children, animals, vehicles and others; The target recognition result correction module is used to correct the target type of the second recognition target set based on the first recognition target set.

15. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 13; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 13.

Images