A method and device for end-to-end dependency structure analysis based on speech

Abstract

The present invention relates to the field of natural language processing technology, and in particular to a method and device for end-to-end dependency structure analysis based on speech. The method comprises: obtaining feature vectors from speech data based on a speech feature extractor, inputting the feature vectors into a speech decoder to obtain a speech embedding vector representation; mapping the speech embedding vector representation to a text space based on an adapter; outputting the speech dependency structure analysis results of the mapped data through a dual affine decoder; label-aligning the transcribed text to obtain the alignment labels of the transcribed text; and training a speech dependency structure analysis model end-to-end through the alignment labels. The present invention directly starts from speech to realize the utilization of speech features, uses an alignment tool to map the label information of the gold label text to the transcribed text, and fully utilizes the dependency structure analysis data to train the speech dependency structure analysis model.

Claims

1. A method for end-to-end dependency structure analysis based on speech, characterized in that: The method comprises: S1. Acquire speech data, obtain a speech feature vector through a speech feature extractor, input the speech feature vector into a speech decoder, and obtain a speech embedding vector representation; S2. Mapping the speech embedding vector representation to the text space based on the adapter to obtain the text embedding vector representation; The adapter-based mapping of the speech embedding vector representation to the text space in S2 to obtain the text embedding vector representation includes: Preset adapter; the adapter is implemented by constructing a mapping matrix from the speech decoder vocabulary to the bi-affine decoder vocabulary; The speech embedding vector is represented as the corresponding one-hot vector, and the one-hot vector is multiplied by the mapping matrix through matrix multiplication to obtain the text embedding vector representation; Mapping the speech embedding vector representation to the text space through the adapter to obtain a text embedding vector representation corresponding to the speech embedding vector representation; S3. Decoding the mapped text embedding vector representation to obtain a transcribed text, and performing label alignment on the transcribed text to obtain an aligned label of the transcribed text; S4. The text is embedded in the vector representation and the dependency structure analysis result of the speech is outputted through a dual affine decoder; the speech dependency structure analysis model is trained end-to-end using the dependency structure analysis result of the speech and the alignment label, and the dependency structure analysis of the speech is completed based on the trained speech dependency structure analysis model.

2. The method for end-to-end speech dependency structure analysis according to claim 1, wherein: In S3, decoding the mapped text embedding vector representation to obtain a transcribed text, performing label alignment on the transcribed text, and obtaining an aligned label of the transcribed text includes: Get the gold standard text labels of the original speech data; The transcribed text is aligned with the gold-labeled text label to obtain an aligned label of the transcribed text.

3. The method for end-to-end speech dependency structure analysis according to claim 2, wherein: The text embedding vector representation in S4 is outputted through the dual affine decoder to analyze the dependency structure of the speech; The method includes: training a speech dependency structure analysis model end-to-end through the alignment labels, and completing speech dependency structure analysis based on the trained speech dependency structure analysis model, including: Inputting the text embedding vector representation into a dual affine decoder to obtain the dependency relationship between words in the text and outputting the dependency structure analysis result of the speech; The speech dependency structure analysis result and the aligned labels are used to calculate the loss to train the speech dependency structure analysis model end-to-end, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

4. A device for end-to-end speech dependency structure analysis, wherein the device is used to implement the method for end-to-end speech dependency structure analysis according to any one of claims 1 to 3, characterized in that: The device comprises: A speech conversion module is used to obtain speech data, obtain speech feature vectors from the speech data through a speech feature extractor, and input the speech feature vectors into a speech decoder to obtain a speech embedding vector representation; A data mapping module, configured to map the speech embedding vector representation to the text space based on the adapter to obtain the text embedding vector representation; A label alignment module decodes the mapped text embedding vector representation to obtain a transcribed text, performs label alignment on the transcribed text, and obtains an alignment label for the transcribed text; The model training module is used to embed the text into a vector representation and output the speech dependency structure analysis result through a dual affine decoder; the speech dependency structure analysis model is trained end-to-end through the alignment label, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

5. A device for end-to-end speech dependency structure analysis, characterized in that: The end-to-end voice dependency structure analysis device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images