A region of interest based audio enhancement encoding transmission system and method

Abstract

The application discloses an audio enhancement coding transmission system and method based on a region of interest, which comprises a microphone array sampling module, a multi-sound source extraction module, a down-sampling module, a neural network interest recognition module, a sound interest selection module, a sound interest coding module, a packing transmission module, a sound interest decoding module, an up-sampling module, a sound selection mixing module and a loudspeaker rendering module. The audio enhancement coding transmission system and method based on the region of interest can adopt high frame rate coding for the sound in the region of interest, improve the sound content information quantity in the region of interest, adopt low frame rate coding for the non-region of interest, and balance network transmission and subsequent data storage data quantity.

Description

TECHNICAL FIELD

[0001] The present application relates to an audio enhancement coding transmission system and method based on a region of interest. BACKGROUND

[0002] The existing sound acquisition device does not distinguish between sounds in different directions, and uniformly encodes the highest sound in the selected sound. The sound in the region of interest and the sound in the region of interest are uniformly processed, and the useless information occupies the transmission and storage resources, and the network transmission and subsequent data storage data volume cannot be balanced. SUMMARY

[0003] The present application provides an audio enhancement coding transmission system and method based on a region of interest to solve the above technical problems, which specifically adopts the following technical scheme:

[0004] An audio enhancement coding transmission system based on a region of interest, comprising: a microphone array sampling module, a multi-sound source extraction module, a down-sampling module, a neural network interest recognition module, a sound interest selection module, a sound interest coding module, a packaging transmission module, a sound interest decoding module, an up-sampling module, a sound selection mixing module and a loudspeaker rendering module;

[0005] The microphone array sampling module collects sound through a plurality of microphones, and each microphone independently outputs high-sampling-rate PCM audio data and a timestamp to the multi-sound source extraction module;

[0006] The multi-sound source extraction module receives the PCM audio data and the timestamp of each sound source sent by the microphone array sampling module, obtains the angle direction of each sound source, uses the difference between the phases of the sound waves received by the plurality of microphones to filter the remaining angle sound waves for each sound source angle, obtains the required high-sampling-rate PCM data of each sound source, obtains the block label according to the angle of each sound source, and sends the high-sampling-rate PCM data, the block label and the timestamp of each sound source to the down-sampling module and the sound interest coding module;

[0007] The down-sampling module performs a low-sampling-rate processing on the high-sampling-rate PCM data of each sound source to obtain first low-sampling-rate PCM data, and sends the first low-sampling-rate PCM data, the block label and the timestamp of each sound source to the neural network interest recognition module. The down-sampling module further performs a low-sampling-rate processing on the first low-sampling-rate PCM data of each sound source to obtain second low-sampling-rate PCM data, and sends the second low-sampling-rate PCM data, the block label and the timestamp of each sound source to the sound interest coding module;

[0008] The neural network interest recognition module receives the first low sampling rate PCM data, the block label and the timestamp of each sound source, converts the voice into text, classifies the text, and determines whether it is the sound source of interest. The neural network interest recognition module sends the classification result, the block label and the timestamp of each sound source to the sound interest selection module;

[0009] The sound interest selection module receives the classification result, the block label and the timestamp of each sound source of the neural network interest recognition module. The sound interest selection module divides the timestamp according to the slice duration T, obtains the time number sequence by dividing the timestamp by the slice duration T, generates the partition time slot ID of each sound source by combining the time number sequence and the block label, and identifies the partition time slot ID that meets the sound source block label and the time number sequence greater than or equal to the current time number sequence and the time number sequence less than or equal to the current time number sequence plus the adjustment width K as the interest selection for the classification result being the sound source of interest. The sound interest selection module sends the partition time slot ID identified as the interest selection to the sound interest encoding module.

[0010] The sound interest encoding module receives the high sampling rate PCM data, the block label and the timestamp of each sound source sent by the multi-sound source extraction module, receives the second low sampling rate PCM data, the block label and the timestamp of each sound source sent by the down-sampling module, and receives the partition time slot ID selected as the interest selection sent by the sound interest selection module. The sound interest encoding module divides the high sampling rate timestamp by the slice duration T to obtain the time number sequence, generates the partition time slot ID of the high sampling rate by combining the time number sequence and the block label, divides the low sampling rate timestamp by the slice duration T to obtain the time number sequence, generates the partition time slot ID of the low sampling rate by combining the time number sequence and the block label, selects the high sampling rate PCM data for audio encoding compression to obtain the audio enhanced compression stream of the sound source for the partition time slot ID being the interest selection slice, selects the second low sampling rate PCM data for audio encoding compression to obtain the audio standard compression stream of the sound source for the partition time slot ID not being the interest selection slice, and sends the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the packaging transmission module.

[0011] The packaging transmission module receives the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID, performs network RTP packaging, and sends it to the sound interest decoding module.

[0012] The sound interest decoding module receives the audio standard compression stream or the audio enhancement compression stream of each sound source and the corresponding partition time slot ID sent by the packet transmission module, decodes the audio compression stream to obtain high sampling rate PCM data and second low sampling rate PCM data, sends the high sampling rate PCM data of each sound source and the partition time slot ID to the sound selection and mixing module, and sends the second low sampling rate PCM data of each sound source and the partition time slot ID to the upsampling module.

[0013] The upsampling module performs upsampling on the second low sampling rate PCM data to obtain upsampling PCM data, and sends the upsampling PCM data of each sound source and the partition time slot ID to the sound selection and mixing module.

[0014] The sound selection and mixing module receives the high sampling rate PCM data of each sound source and the partition time slot ID sent by the sound interest decoding module and the upsampling PCM data of each sound source and the partition time slot ID sent by the upsampling module, selects the sound source that needs to be mixed according to the set time interval and direction interval, divides the current timestamp or the selected playing time by the slice time length T to obtain a playing time number sequence, obtains a block label list that meets the direction interval according to the playing time number sequence, combines the block label in the block label list and the time number sequence to obtain the partition time slot ID, obtains a partition time slot ID list that allows mixing, judges whether the partition time slot ID of each sound source received is in the partition time slot ID list, and if yes, participates in mixing, performs mixing calculation on the PCM data of the sound source participating in mixing to obtain mixed PCM data, and sends the mixed PCM data to the loudspeaker rendering module.

[0015] The loudspeaker rendering module receives the mixed PCM data of the sound selection and mixing module, performs digital-to-analog conversion, and plays the mixed sound.

[0016] Further, the multi-sound source extraction module obtains the angle direction of each sound source according to the direction of arrival estimation algorithm, divides the horizontal angle into 12 blocks according to 30 degrees, labels each block, and obtains the block label according to the angle of each sound source.

[0017] Further, the downsampling module reduces the 96KHz high sampling rate PCM data to 16KHz first low sampling rate PCM data, and reduces the 16KHz first low sampling rate PCM data to 8KHz second low sampling rate PCM data.

[0018] Further, the neural network interest recognition module converts the speech into text using a GMM-HMM based acoustic model;

[0019] The interest word is data-extended by a pre-trained word vector with semantic knowledge, and the interest word classification model is obtained by training the CNN neural network model based on the interest word corpus, data extension margin, and non-interest word corpus. The neural network interest recognition module classifies the converted text through the trained interest word classification model.

[0020] Further, the upsampling module upsamples the 8KHz second low sampling rate PCM data to 96KHz upsampled PCM data.

[0021] An audio enhancement encoding transmission system based on a region of interest includes the following steps:

[0022] The microphone array sampling module collects sound through multiple microphones, and each microphone independently outputs high sampling rate PCM audio data and a timestamp to the multi-source extraction module;

[0023] The multi-source extraction module receives the PCM audio data and timestamp of each sound source sent by the microphone array sampling module, obtains the angular orientation of each sound source, and uses the difference between the phases of the sound waves received by multiple microphones to filter the remaining angular sound waves for each sound source angle to obtain the required high sampling rate PCM data of each sound source. According to the angle of each sound source, the block label is obtained, and the high sampling rate PCM data, block label, and timestamp of each sound source are sent to the downsampling module and the sound interest encoding module;

[0024] The downsampling module performs low sampling rate processing on the high sampling rate PCM data of each sound source to obtain first low sampling rate PCM data, and sends the first low sampling rate PCM data, block label, and timestamp of each sound source to the neural network interest recognition module. The downsampling module further performs low sampling rate processing on the first low sampling rate PCM data of each sound source to obtain second low sampling rate PCM data, and sends the second low sampling rate PCM data, block label, and timestamp of each sound source to the sound interest encoding module;

[0025] The neural network interest recognition module receives the first low sampling rate PCM data, block label, and timestamp of each sound source, converts the speech into text, classifies the text, and determines whether it is an interest sound source. The neural network interest recognition module sends the classification result, block label, and timestamp of each sound source to the sound interest selection module;

[0026] The sound interest selection module receives the classification result of each sound source of the neural network interest recognition module, the block label and the timestamp, the sound interest selection module divides the timestamp according to the slice duration T, divides the timestamp by the slice duration T to obtain a time number sequence, generates a partition time slot ID of each sound source in combination with the time number sequence and the block label, and identifies the partition time slot ID that meets the sound source to the block label and the time number sequence greater than or equal to the current time number sequence and the time number sequence less than or equal to the current time number sequence plus the adjustment width K as the interest selection for the classification result being the sound source of interest, and sends the partition time slot ID identified as the interest selection to the sound interest coding module.

[0027] The sound interest coding module receives the high-sampling-rate PCM data of each sound source, the block label and the timestamp sent by the multi-sound-source extraction module, receives the second low-sampling-rate PCM data of each sound source, the block label and the timestamp sent by the downsampling module, and receives the partition time slot ID of the interest selection sent by the sound interest selection module. The sound interest coding module divides the high-sampling-rate timestamp by the slice duration T to obtain a time number sequence, generates a high-sampling-rate partition time slot ID in combination with the time number sequence and the block label, divides the low-sampling-rate timestamp by the slice duration T to obtain a time number sequence, and generates a low-sampling-rate partition time slot ID in combination with the time number sequence and the block label. For the partition time slot ID being the interest selection slice, the sound interest coding module selects the high-sampling-rate PCM data for audio encoding compression to obtain an audio enhanced compression stream of the sound source, and for the partition time slot ID not being the interest selection slice, the sound interest coding module selects the second low-sampling-rate PCM data for audio encoding compression to obtain an audio standard compression stream of the sound source. The sound interest coding module sends the audio standard compression stream or the audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the packet transmission module.

[0028] The packet transmission module receives the audio standard compression stream or the audio enhanced compression stream of each sound source and the corresponding partition time slot ID, performs network RTP packeting, and sends the same to the sound interest decoding module.

[0029] The sound interest decoding module receives the audio standard compression stream or the audio enhanced compression stream of each sound source and the corresponding partition time slot ID sent by the packet transmission module. The sound interest decoding module performs audio decoding on the audio compression stream to obtain high-sampling-rate PCM data and second low-sampling-rate PCM data, sends the high-sampling-rate PCM data of each sound source and the partition time slot ID to the sound selection mixing module, and sends the second low-sampling-rate PCM data of each sound source and the partition time slot ID to the upsampling module.

[0030] The upsampling module performs upsampling processing on the second low sampling rate PCM data to obtain upsampling PCM data, and sends the upsampling PCM data and the partition time slot ID of each sound source to the sound selection and mixing module.

[0031] The sound selection and mixing module receives the high sampling rate PCM data and the partition time slot ID of each sound source sent from the sound of interest decoding module, and the upsampling PCM data and the partition time slot ID of each sound source sent from the upsampling module. The sound selection and mixing module selects the sound source that needs to be mixed according to the set time interval and direction interval. The sound selection and mixing module divides the current timestamp or the selected playback time by the slice duration T to obtain a playback time number sequence, obtains a block label list corresponding to the direction interval, and obtains the partition time slot ID by combining the block label in the block label list and the time number sequence. The sound selection and mixing module obtains a partition time slot ID list that allows mixing. The sound selection and mixing module judges whether the partition time slot ID of each sound source received is in the partition time slot ID list. If yes, the sound source participates in mixing. The sound selection and mixing module performs mixing calculation on the PCM data of the sound source participating in mixing to obtain mixed PCM data, and sends the mixed PCM data to the loudspeaker rendering module.

[0032] The loudspeaker rendering module receives the mixed PCM data from the sound selection and mixing module, performs digital-to-analog conversion, and plays the mixed sound.

[0033] Further, the specific method of the multi-sound source extraction module for obtaining the block label is as follows:

[0034] The multi-sound source extraction module obtains the angle direction of each sound source according to the direction of arrival estimation algorithm. The multi-sound source extraction module divides the horizontal angle into 12 blocks according to 30 degrees, labels each block, and obtains the block label according to the angle of each sound source.

[0035] Further, the specific method of the downsampling module for performing the downsampling processing is as follows:

[0036] The downsampling module reduces the 96KHz high sampling rate PCM data to 16KHz first low sampling rate PCM data, and then reduces the 16KHz first low sampling rate PCM data to 8KHz second low sampling rate PCM data.

[0037] Further, the specific method of the neural network of interest recognition module for converting speech into text is as follows:

[0038] The neural network of interest recognition module converts speech into text using an acoustic model based on GMM-HMM.

[0039] The specific method for the neural network interest recognition module to classify the converted text is as follows:

[0040] The interested word is expanded by the pre-trained word vector with semantic knowledge, the CNN neural network model is trained by the interested word corpus, the data expansion margin and the non-interested word corpus to obtain an interested word classification model, and the neural network interest recognition module classifies the converted text by using the trained interested word classification model.

[0041] Further, the specific method for the upsampling module to perform upsampling processing is as follows:

[0042] The upsampling module upsamples the 8KHz second low sampling rate PCM data to 96KHz upsampling PCM data.

[0043] The present application has the advantages that the provided audio enhancement encoding transmission system and method based on the interested region adopt high frame rate encoding for the sound of the interested region, improve the sound content information amount of the interested region, adopt low frame rate encoding for the non-interested region, and balance the network transmission and subsequent data storage data amount. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a schematic diagram of an audio enhancement encoding transmission system based on the interested region of the present application. DETAILED DESCRIPTION

[0045] The present application will be specifically introduced below in combination with the drawings and specific embodiments.

[0046] As Figure 1 It is an audio enhancement encoding transmission system based on the interested region of the present application, which comprises a microphone array sampling module 1, a multi-sound source extraction module 2, a downsampling module 3, a neural network interest recognition module 4, a sound interest selection module 5, a sound interest encoding module 6, a packaging transmission module 7, a sound interest decoding module 8, an upsampling module 9, a sound selection mixing module 10 and a loudspeaker rendering module 11.

[0047] Among them, the microphone array sampling module 1 collects sound through multiple microphones, and each microphone independently outputs high sampling rate PCM audio data and a timestamp to the multi-sound source extraction module 2. In the present application, the high sampling rate PCM audio data is 96KHz.

[0048] The multi-sound source extraction module 2 is configured to extract PCM audio data of multi-sound sources of the microphone array. Specifically, the multi-sound source extraction module 2 receives the PCM audio data and the time stamp of each sound source sent by the microphone array sampling module 1, obtains the angular orientation of each sound source, filters the sound waves of the remaining angles using the difference between the phases of the sound waves received by the plurality of microphones for each angle of each sound source, respectively, to obtain the high sampling rate PCM data of each sound source needed. According to the block number of each sound source, the high sampling rate PCM data, the block number and the time stamp of each sound source are sent to the down-sampling module 3 and the sound of interest coding module 6.

[0049] Specifically, the multi-sound source extraction module 2 obtains the angular orientation of each sound source according to the Direction-of-Arrival (DOA) algorithm. Then, the multi-sound source extraction module 2 divides the horizontal angle into 12 blocks according to 30 degrees, and labels each block, and obtains the block number of each sound source according to the angle.

[0050] The down-sampling module 3 is responsible for reducing the sampling rate of the PCM data of each sound source of the multi-sound source extraction module 2, for reducing the amount of data calculation and the amount of data transmission and packaging during recognition analysis. Specifically, the down-sampling module 3 reduces the sampling rate of the high sampling rate PCM data of each sound source to obtain first low sampling rate PCM data, and sends the first low sampling rate PCM data, the block number and the time stamp of each sound source to the neural network of interest recognition module 4. Specifically, the down-sampling module 3 further reduces the sampling rate of the first low sampling rate PCM data of each sound source to obtain second low sampling rate PCM data, and sends the second low sampling rate PCM data, the block number and the time stamp of each sound source to the sound of interest coding module 6. In this application, the down-sampling module 3 reduces the high sampling rate PCM data of 96KHz to the first low sampling rate PCM data of 16KHz, and then reduces the first low sampling rate PCM data of 16KHz to the second low sampling rate PCM data of 8KHz. Specifically, the down-sampling module 3 uses decimation method to reduce the frame rate. The down-sampling module 3 converts the high sampling rate PCM data of each sound source to 16KHZ. The PCM data has a period of 6 sampling points, and only the first sampling point is retained in each period to obtain 16KHZ PCM data. The down-sampling module 3 further converts the 16KHZ data to 8KHZ, and the PCM data has a period of 2 sampling points, and only the first sampling point is retained in each period to obtain 8KHZ low sampling audio.

[0051] The neural network interest recognition module 4 is mainly used to determine the content of interest of each sound source. The neural network interest recognition module 4 receives the first low sampling rate PCM data, the block label and the timestamp of each sound source, converts the voice into text, classifies the text, and determines whether it is a sound source of interest. The neural network interest recognition module 4 sends the classification result, the block label and the timestamp of each sound source to the sound interest selection module 5.

[0052] Specifically, the neural network interest recognition module 4 converts the voice into text using a GMM-HMM-based acoustic model. The neural network interest recognition module 4 performs data expansion on the interested words by using a pre-trained word vector with semantic knowledge, trains a CNN neural network model using interested word corpus, data expansion margin and non-interested word corpus to obtain an interested word classification model, and classifies the converted text by using the trained interested word classification model.

[0053] The sound interest selection module 5 controls the use of high frame rate encoding for each sound source in each time slice according to whether it contains content of interest. Specifically, the sound interest selection module 5 receives the classification result, the block label and the timestamp of each sound source from the neural network interest recognition module 4. The sound interest selection module 5 divides the timestamp into slices according to the slice duration T (for example, 500 ms), divides the timestamp by the slice duration T to obtain the time number sequence, and generates the partition time slot ID of each sound source by combining the time number sequence and the block label. For the sound source with the classification result of interest, the partition time slot ID that meets the condition of sound source to block label and time number sequence greater than or equal to the current time number sequence and time number sequence less than or equal to the current time number sequence plus the adjustment width K (for example, 40) is identified as the interested selection, and the partition time slot ID identified as the interested selection is sent to the sound interest encoding module 6.

[0054] The sound interest coding module 6 selectively high frame rate encodes according to the output of the sound interest selection module 5. The sound interest coding module 6 receives the high sampling rate PCM data, the tile index and the time stamp of each sound source sent by the multi-sound source extraction module 2, receives the second low sampling rate PCM data, the tile index and the time stamp of each sound source sent by the down-sampling module 3, and receives the selected partition time slot ID sent by the sound interest selection module 5. The sound interest coding module 6 divides the time stamp of the high sampling rate by the tile duration T to obtain a time number sequence, and generates the partition time slot ID of the high sampling rate in combination with the time number sequence and the tile index. The sound interest coding module 6 divides the time stamp of the low sampling rate by the tile duration T to obtain a time number sequence, and generates the partition time slot ID of the low sampling rate in combination with the time number sequence and the tile index. For the selected partition time slot ID, the high sampling rate PCM data is selected for audio encoding compression to obtain the audio enhanced compression stream of the sound source, and for the non-selected partition time slot ID, the second low sampling rate PCM data is selected for audio encoding compression to obtain the audio standard compression stream of the sound source. The sound interest coding module 6 sends the audio standard compression stream or the audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the packet transmission module 7.

[0055] The packet transmission module 7 receives the audio standard compression stream or the audio enhanced compression stream of each sound source and the corresponding partition time slot ID, performs network RTP packetization, and sends to the sound interest decoding module 8.

[0056] The sound interest decoding module 8 receives the audio standard compression stream or the audio enhanced compression stream of each sound source and the corresponding partition time slot ID sent by the packet transmission module 7. The sound interest decoding module 8 performs audio decoding on the audio compression stream to obtain the high sampling rate PCM data and the second low sampling rate PCM data, sends the high sampling rate PCM data of each sound source and the partition time slot ID to the sound selection mixing module 10, and sends the second low sampling rate PCM data of each sound source and the partition time slot ID to the up-sampling module 9.

[0057] The upsampling module 9 performs upsampling processing on the second low sampling rate PCM data to obtain upsampling PCM data, and sends the upsampling PCM data of each sound source and the partition time slot ID to the sound selection and mixing module 10. Specifically, the upsampling module 9 performs interpolation calculation on the second low sampling rate PCM data, such as upsampling the second low sampling rate PCM data to 96KHz upsampling PCM data, inserting 11 PCM data using a linear method between two adjacent points of the PCM data, and filtering the upsampling PCM data using an 8KHz low-pass filter to remove high-frequency interference introduced due to interpolation. The upsampling module 9 sends the processed upsampling PCM data and the partition time slot ID to the sound selection and mixing module 10.

[0058] The sound selection and mixing module 10 receives the high sampling rate PCM data of each sound source and the partition time slot ID sent from the sound of interest decoding module 8, and the upsampling PCM data of each sound source and the partition time slot ID sent from the upsampling module 9. The sound selection and mixing module 10 selects the sound source to be mixed according to the set time interval and direction interval.

[0059] The sound selection and mixing module 10 selects the sound to be mixed according to time and direction. Specifically, the sound selection and mixing module 10 divides the current timestamp or the selected playback time by the slice duration T to obtain a playback time number sequence, obtains a block label list that meets the direction interval according to the playback time number sequence, combines the block label in the block label list with the time number sequence to obtain the partition time slot ID, and obtains a list of partition time slot IDs that are allowed to be mixed. The sound selection and mixing module 10 judges whether the partition time slot ID of each sound source received is in the list of partition time slot IDs. If yes, the sound source participates in mixing. The sound selection and mixing module 10 performs mixing calculation on the PCM data of the sound source participating in mixing to obtain mixed PCM data, and sends the mixed PCM data to the loudspeaker rendering module 11.

[0060] The loudspeaker rendering module 11 receives the mixed PCM data from the sound selection and mixing module 10, performs digital-to-analog conversion, and plays the mixed sound.

[0061] The application also discloses an audio enhancement encoding transmission method based on a region of interest, comprising the following steps:

[0062] The microphone array sampling module 1 collects sound through multiple microphones, and each microphone independently outputs high sampling rate PCM audio data and a timestamp to the multi-sound source extraction module 2.

[0063] The multi-sound source extraction module 2 receives the PCM audio data and the time stamp of each sound source sent by the microphone array sampling module 1, obtains the angular direction of each sound source, filters the sound waves of the remaining angles using the difference between the phases of the sound waves received by the multiple microphones for each angular direction of each sound source, obtains the high sampling rate PCM data of each sound source, and sends the high sampling rate PCM data, the block label and the time stamp of each sound source to the down-sampling module 3 and the sound of interest coding module 6 according to the angular direction of each sound source.

[0064] The down-sampling module 3 performs a down-sampling rate processing on the high sampling rate PCM data of each sound source to obtain first low sampling rate PCM data, and sends the first low sampling rate PCM data, the block label and the time stamp of each sound source to the neural network of interest recognition module 4. The down-sampling module 3 further performs a down-sampling rate processing on the first low sampling rate PCM data of each sound source to obtain second low sampling rate PCM data, and sends the second low sampling rate PCM data, the block label and the time stamp of each sound source to the sound of interest coding module 6.

[0065] The neural network of interest recognition module 4 receives the first low sampling rate PCM data, the block label and the time stamp of each sound source, converts the voice into text, classifies the text, and determines whether it is a sound source of interest. The neural network of interest recognition module 4 sends the classification result, the block label and the time stamp of each sound source to the sound of interest selection module 5.

[0066] The sound of interest selection module 5 receives the classification result, the block label and the time stamp of each sound source from the neural network of interest recognition module 4. The sound of interest selection module 5 divides the time stamp into fragments according to the fragment duration T, obtains a time number sequence by dividing the time stamp by the fragment duration T, generates a partition time slot ID of each sound source by combining the time number sequence and the block label, and identifies the partition time slot ID of the sound source of interest as a sound of interest selection if the classification result is a sound source of interest, and sends the partition time slot ID of the sound of interest selection to the sound of interest coding module 6.

[0067] The sound interest coding module 6 receives the high sampling rate PCM data, the block label and the time stamp of each sound source sent by the multi-sound source extraction module 2, receives the second low sampling rate PCM data, the block label and the time stamp of each sound source sent by the down-sampling module 3, receives the selected partition time slot ID sent by the sound interest selection module 5, divides the time stamp of the high sampling rate by the time slice length T to obtain a time number sequence, generates the partition time slot ID of the high sampling rate by combining the time number sequence and the block label, divides the time stamp of the low sampling rate by the time slice length T to obtain a time number sequence, generates the partition time slot ID of the low sampling rate by combining the time number sequence and the block label, selects the high sampling rate PCM data for audio encoding compression to obtain the audio enhanced compression stream of the sound source for the selected partition time slot ID, selects the second low sampling rate PCM data for audio encoding compression to obtain the audio standard compression stream of the sound source for the non-selected partition time slot ID, and sends the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the packet transmission module 7.

[0068] The packet transmission module 7 receives the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID, performs network RTP packet transmission, and sends the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the sound interest decoding module 8.

[0069] The sound interest decoding module 8 receives the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID sent by the packet transmission module 7, performs audio decoding on the audio compression stream to obtain the high sampling rate PCM data and the second low sampling rate PCM data, sends the high sampling rate PCM data of each sound source and the partition time slot ID to the sound selection mixing module 10, and sends the second low sampling rate PCM data of each sound source and the partition time slot ID to the up-sampling module 9.

[0070] The up-sampling module 9 performs up-sampling processing on the second low sampling rate PCM data to obtain up-sampled PCM data, and sends the up-sampled PCM data of each sound source and the partition time slot ID to the sound selection mixing module 10.

[0071] The sound selection and mixing module 10 receives the high sampling rate PCM data of each sound source sent from the sound source extraction module 2, the partition time slot ID sent from the sound source extraction module 2, and the down-sampled PCM data of each sound source sent from the down-sampling module 3 and the partition time slot ID sent from the down-sampling module 3. The sound selection and mixing module 10 selects the sound sources that need to be mixed according to the set time interval and direction interval. The sound selection and mixing module 10 divides the current time stamp or the selected playing time by the slice time T to obtain a playing time number sequence, obtains a block label list in which the block labels conform to the direction interval, and obtains the partition time slot ID by combining the block labels in the block label list and the time number sequence. The sound selection and mixing module 10 obtains a list of partition time slot IDs that are allowed to be mixed. The sound selection and mixing module 10 judges whether the partition time slot ID of each sound source received is in the list of partition time slot IDs. If yes, the sound source participates in mixing. The sound selection and mixing module 10 performs mixing calculation on the PCM data of the sound sources participating in mixing to obtain mixed PCM data, and sends the mixed PCM data to the loudspeaker rendering module 11.

[0072] The loudspeaker rendering module 11 receives the mixed PCM data of the sound selection and mixing module 10, performs digital-to-analog conversion, and plays the mixed sound.

[0073] As a preferred embodiment, the specific method of the multi-sound source extraction module 2 for obtaining the block label is as follows:

[0074] The multi-sound source extraction module 2 obtains the angle direction of each sound source according to the direction of arrival estimation algorithm. The multi-sound source extraction module 2 divides the horizontal angle into 12 blocks according to 30 degrees, labels each block, and obtains the block label according to the angle of each sound source.

[0075] As a preferred embodiment, the specific method of the down-sampling module 3 for performing the down-sampling rate processing is as follows:

[0076] The down-sampling module 3 reduces the high sampling rate PCM data of 96 KHz to the first low sampling rate PCM data of 16 KHz, and then reduces the first low sampling rate PCM data of 16 KHz to the second low sampling rate PCM data of 8 KHz.

[0077] As a preferred embodiment, the specific method of the neural network interest recognition module 4 for converting the speech into text is as follows:

[0078] The neural network interest recognition module 4 converts the speech into text using an acoustic model based on GMM-HMM.

[0079] The specific method of the neural network interest recognition module 4 for classifying the converted text is as follows:

[0080] The interested word is data-extended by the pre-trained word vector with semantic knowledge, the CNN neural network model is trained by the interested word corpus, the data extension margin and the non-interested word corpus to obtain an interested word classification model, and the neural network interested recognition module 4 classifies the converted text by using the trained interested word classification model.

[0081] As a preferred embodiment, the specific method of the upsampling module 9 for upsampling processing is as follows:

[0082] The upsampling module 9 upsamples the second low sampling rate PCM data of 8KHz to the upsampling PCM data of 96KHz.

[0083] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the above embodiments do not limit the present application in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the protection scope of the present application.

Claims

1. A region of interest based audio enhancement encoding transmission system, characterized in that, The system comprises a microphone array sampling module, a multi-sound source extraction module, a down-sampling module, a neural network interest identification module, a sound interest selection module, a sound interest encoding module, a packaging and transmission module, a sound interest decoding module, an up-sampling module, a sound selection mixing module, and a loudspeaker rendering module. The microphone array sampling module collects sound through multiple microphones, and each microphone independently outputs high-sampling-rate PCM audio data and a timestamp to the multi-sound source extraction module. The multi-sound source extraction module receives the PCM audio data and the timestamp of each sound source sent by the microphone array sampling module, obtains the angular orientation of each sound source, filters the sound waves of the remaining angles using the difference between the phases of the sound waves received by multiple microphones for each angular orientation of each sound source, obtains the high-sampling-rate PCM data of each sound source that needs to be obtained, obtains the block label according to the angular orientation of each sound source, and sends the high-sampling-rate PCM data, the block label, and the timestamp of each sound source to the down-sampling module and the sound interest encoding module. The down-sampling module performs a low-sampling-rate processing on the high-sampling-rate PCM data of each sound source to obtain first low-sampling-rate PCM data, sends the first low-sampling-rate PCM data, the block label, and the timestamp of each sound source to the neural network interest identification module, and performs a low-sampling-rate processing on the first low-sampling-rate PCM data of each sound source to obtain second low-sampling-rate PCM data, and sends the second low-sampling-rate PCM data, the block label, and the timestamp of each sound source to the sound interest encoding module. The neural network interest identification module receives the first low-sampling-rate PCM data, the block label, and the timestamp of each sound source, converts speech into text, classifies the text, and determines whether it is a sound source of interest. The neural network interest identification module sends the classification result, the block label, and the timestamp of each sound source to the sound interest selection module. The sound interest selection module receives the classification result, the block label, and the timestamp of each sound source from the neural network interest identification module, divides the timestamp into fragments according to the fragment duration T, obtains a time number sequence by dividing the timestamp by the fragment duration T, generates a partition time slot ID of each sound source by combining the time number sequence and the block label, and identifies the partition time slot ID that meets the sound source-to-block label and the time number sequence greater than or equal to the current time number sequence and less than or equal to the current time number sequence plus the adjustment width K as a sound source of interest for the classification result of the sound source of interest, and sends the partition time slot ID identified as the sound source of interest to the sound interest encoding module. The sound interest coding module receives the high sampling rate PCM data, the block label and the time stamp of each sound source sent by the multi-sound source extraction module, receives the second low sampling rate PCM data, the block label and the time stamp of each sound source sent by the down-sampling module, receives the selected partition time slot ID of the interest selection sent by the sound interest selection module, divides the time stamp of the high sampling rate by the time slice length T to obtain a time number sequence, generates the partition time slot ID of the high sampling rate by combining the time number sequence and the block label, divides the time stamp of the low sampling rate by the time slice length T to obtain a time number sequence, generates the partition time slot ID of the low sampling rate by combining the time number sequence and the block label, selects the high sampling rate PCM data for audio encoding compression to obtain the audio enhanced compression stream of the sound source for the partition time slot ID of the selected slice of interest, selects the second low sampling rate PCM data for audio encoding compression to obtain the audio standard compression stream of the sound source for the partition time slot ID of the slice of interest which is not selected, and sends the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the packet transmission module; The packet transmission module receives the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID, performs network RTP packeting, and sends to the sound interest decoding module; The sound interest decoding module receives the audio standard compression stream or audio enhanced compression stream of each sound source and the corresponding partition time slot ID sent by the packet transmission module, performs audio decoding on the audio compression stream to obtain the high sampling rate PCM data and the second low sampling rate PCM data, sends the high sampling rate PCM data of each sound source and the partition time slot ID to the sound selection mixing module, and sends the second low sampling rate PCM data of each sound source and the partition time slot ID to the up-sampling module; The up-sampling module performs up-sampling processing on the second low sampling rate PCM data to obtain up-sampled PCM data, and sends the up-sampled PCM data of each sound source and the partition time slot ID to the sound selection mixing module. The sound selection and mixing module receives the high sampling rate PCM data of each sound source sent from the sound of interest decoding module and the partition time slot ID, and the PCM data after up-sampling of each sound source sent from the up-sampling module and the partition time slot ID. The sound selection and mixing module selects the sound source that needs to be mixed according to the set time interval and direction interval. The sound selection and mixing module divides the current timestamp or the selected playing time by the slice time length T to obtain a playing time number sequence, obtains a block label list in which the playing time number sequence meets the direction interval, combines the block label in the block label list with the time number sequence to obtain the partition time slot ID, obtains a partition time slot ID list that allows mixing, and judges whether the partition time slot ID of each sound source received by the sound selection and mixing module is in the partition time slot ID list. If yes, the sound source participates in mixing. The sound selection and mixing module performs mixing calculation on the PCM data of the sound source participating in mixing to obtain mixed PCM data, and sends the mixed PCM data to the loudspeaker rendering module. The loudspeaker rendering module receives the mixed PCM data of the sound selection and mixing module, performs digital-to-analog conversion, and plays the mixed sound.

2. The audio enhancement coding and transmission system based on a region of interest according to claim 1, wherein The multi-sound source extraction module obtains the angular orientation of each sound source according to a direction of arrival estimation algorithm. The multi-sound source extraction module divides the horizontal angle into 12 blocks according to 30 degrees, and labels each block. The block label is obtained according to the angle of each sound source.

3. The audio enhancement coding and transmission system based on a region of interest according to claim 1, wherein The down-sampling module reduces the high sampling rate PCM data of 96 KHz to the first low sampling rate PCM data of 16 KHz, and then reduces the first low sampling rate PCM data of 16 KHz to the second low sampling rate PCM data of 8 KHz.

4. The audio enhancement coding and transmission system based on a region of interest according to claim 1, wherein The neural network interest recognition module converts speech into text using an acoustic model based on GMM-HMM. The interest word is expanded through a pre-trained word vector with semantic knowledge. The CNN neural network model is trained through the interest word corpus, the data expansion margin, and the non-interest word corpus to obtain an interest word classification model. The neural network interest recognition module classifies the converted text through the trained interest word classification model.

5. The audio enhancement coding and transmission system based on a region of interest according to claim 1, wherein The up-sampling module increases the second low sampling rate PCM data of 8 KHz to the PCM data after up-sampling of 96 KHz.

6. A region of interest based audio enhancement encoding transmission method, characterized in that, The method comprises the following steps: The microphone array sampling module collects sound through multiple microphones. Each microphone independently outputs high sampling rate PCM audio data and a timestamp to the multi-sound source extraction module. The multi-sound source extraction module receives the PCM audio data and the time stamp of each sound source sent by the microphone array sampling module, obtains the angle direction of each sound source, respectively uses the difference between the phases of the sound waves received by the multiple microphones to filter the sound waves of the remaining angles for each angle of the sound source, obtains the high sampling rate PCM data of each sound source needed, obtains the block label according to the angle of each sound source, and sends the high sampling rate PCM data, the block label and the time stamp of each sound source to the down-sampling module and the sound of interest coding module; The down-sampling module performs the down-sampling rate processing on the high sampling rate PCM data of each sound source to obtain the first low sampling rate PCM data, sends the first low sampling rate PCM data, the block label and the time stamp of each sound source to the neural network of interest recognition module, and performs the down-sampling rate processing on the first low sampling rate PCM data of each sound source again to obtain the second low sampling rate PCM data, and sends the second low sampling rate PCM data, the block label and the time stamp of each sound source to the sound of interest coding module; The neural network of interest recognition module receives the first low sampling rate PCM data, the block label and the time stamp of each sound source, converts the voice into text, classifies the text, and judges whether it is a sound source of interest, and sends the classification result, the block label and the time stamp of each sound source to the sound of interest selection module; The sound of interest selection module receives the classification result, the block label and the time stamp of each sound source of the neural network of interest recognition module, divides the time stamp according to the fragmentation time length T, obtains the time number sequence by dividing the time stamp by the fragmentation time length T, generates the partition time slot ID of each sound source in combination with the time number sequence and the block label, and identifies the partition time slot ID that meets the sound source to the block label and the time number sequence greater than or equal to the current time number sequence and the time number sequence less than or equal to the current time number sequence plus the adjustment width K as the interest selection for the classification result being the sound source of interest, and sends the partition time slot ID identified as the interest selection to the sound of interest coding module; The sound interest coding module receives the high sampling rate PCM data, the block label and the time stamp of each sound source sent by the multi-sound source extraction module, receives the second low sampling rate PCM data, the block label and the time stamp of each sound source sent by the down-sampling module, receives the selected partition time slot ID of the interest selection sent by the sound interest selection module, divides the time stamp of the high sampling rate by the time slice length T to obtain a time number sequence, generates the partition time slot ID of the high sampling rate by combining the time number sequence and the block label, divides the time stamp of the low sampling rate by the time slice length T to obtain a time number sequence, generates the partition time slot ID of the low sampling rate by combining the time number sequence and the block label, selects the high sampling rate PCM data for audio encoding compression to obtain the audio enhanced compression stream of the sound source for the partition time slot ID of the selected slice of interest, selects the second low sampling rate PCM data for audio encoding compression to obtain the audio standard compression stream of the sound source for the partition time slot ID of the slice of interest which is not selected, and sends the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID to the packet transmission module; The packet transmission module receives the audio standard compression or audio enhanced compression stream of each sound source and the corresponding partition time slot ID, performs network RTP packeting, and sends to the sound interest decoding module; The sound interest decoding module receives the audio standard compression stream or audio enhanced compression stream of each sound source and the corresponding partition time slot ID sent by the packet transmission module, performs audio decoding on the audio compression stream to obtain the high sampling rate PCM data and the second low sampling rate PCM data, sends the high sampling rate PCM data of each sound source and the partition time slot ID to the sound selection mixing module, and sends the second low sampling rate PCM data of each sound source and the partition time slot ID to the up-sampling module; The up-sampling module performs up-sampling processing on the second low sampling rate PCM data to obtain up-sampled PCM data, and sends the up-sampled PCM data of each sound source and the partition time slot ID to the sound selection mixing module. The sound selection and mixing module receives high sampling rate PCM data of each sound source sent from the sound of interest decoding module and partition time slot ID and PCM data of each sound source sent from the upsampling module after upsampling and partition time slot ID, selects sound sources that need to be mixed according to a set time interval and a direction interval, divides the current timestamp or the selected playing time by the slice duration T to obtain a playing time number sequence, obtains a block label list in which the playing time number sequence meets the direction interval, combines the block label in the block label list with the time number sequence to obtain the partition time slot ID, obtains a partition time slot ID list that allows mixing, judges whether the partition time slot ID of each sound source received by the sound selection and mixing module is in the partition time slot ID list, and if so, participates in mixing, and the sound selection and mixing module performs mixing calculation on the PCM data of the sound sources participating in mixing to obtain mixed PCM data, and sends the mixed PCM data to the loudspeaker rendering module; The loudspeaker rendering module receives the mixed PCM data of the sound selection and mixing module, performs digital-to-analog conversion, and plays the mixed sound.

7. The audio enhancement coding transmission method based on a region of interest according to claim 6, characterized in that, The specific method for the multi-sound source extraction module to obtain the block label is that: The multi-sound source extraction module obtains the angle direction of each sound source according to the direction of arrival estimation algorithm, divides the horizontal angle into 12 blocks according to 30 degrees, labels each block, and obtains the block label according to the angle of each sound source.

8. The audio enhancement coding transmission method based on a region of interest according to claim 6, characterized in that, The specific method for the downsampling module to perform the sampling rate reduction processing is that: The downsampling module reduces the high sampling rate PCM data of 96 KHz to the first low sampling rate PCM data of 16 KHz, and then reduces the first low sampling rate PCM data of 16 KHz to the second low sampling rate PCM data of 8 KHz.

9. The audio enhancement coding transmission method based on a region of interest according to claim 6, characterized in that, The specific method for the neural network interest recognition module to convert speech into text is that: The neural network interest recognition module converts speech into text using an acoustic model based on GMM-HMM. The specific method for the neural network interest recognition module to classify the converted text is that: The interest word is expanded through a pre-trained word vector with semantic knowledge, the CNN neural network model is trained through the interest word corpus, the data expansion margin, and the non-interest word corpus to obtain an interest word classification model, and the neural network interest recognition module classifies the converted text through the trained interest word classification model.

10. The audio enhancement coding transmission method based on a region of interest according to claim 6, characterized in that, The specific method for the upsampling module to perform the upsampling processing is that: The up-sampling module up-samples the second low sampling rate PCM data of 8KHz to PCM data of 96KHz after up-sampling. The up-sampling module up-samples the second low sampling rate PCM data of 8KHz to PCM data of 96KHz after

Images