Dependency structure analysis method and device based on voice end-to-end

Abstract

The invention relates to the technical field of natural language processing, in particular to a dependency structure analysis method and device based on voice end-to-end. The method comprises the following steps: acquiring a feature vector from voice data based on a voice feature extractor, and inputting the feature vector into a voice decoder to obtain voice embedded vector representation; mapping the speech embedded vector representation to a text space based on an adapter; outputting a dependency structure analysis result of the voice from the mapped data through a double affine decoder; performing label alignment on the transcriptional text to obtain an aligned label of the transcriptional text; and the model is analyzed through label end-to-end training speech dependency structure alignment. According to the method, the voice features are utilized directly from voice, the label information of the gold label text is mapped to the transcriptional text by using the alignment tool, and the voice dependency structure analysis model is trained by fully utilizing the data of dependency structure analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and device for end-to-end speech-based dependency structure analysis. Background Art

[0002] Dependency structure analysis is a classic task in natural language processing, which aims to identify the dependencies between words in a sentence. Dependency structure analysis can help understand the grammatical structure of a sentence and the relationship between sentence components, which is useful in many downstream natural language processing tasks such as machine translation, information extraction, and question answering.

[0003] At present, research work on dependency structure analysis is mainly focused on text. For speech, there are two mainstream methods: one is the pipeline-based method and the other is the end-to-end method. The pipeline-based method refers to using the text corpus of dependency structure analysis to train a dependency structure analysis model, and combining the existing automatic speech recognition tools to transcribe the speech to obtain text. The text can obtain the dependency structure analysis result for speech through the trained dependency structure model. The pipeline method isolates speech and dependency structure analysis. End-to-end means that the gradient of the speech end of the entire model can be transmitted to the dependency structure analysis end, and the gradient of the dependency structure analysis end can be back-propagated back to the speech end. The end-to-end method can directly use speech signals to train the entire speech dependency structure analysis model. On the one hand, the end-to-end method makes the gradient of the entire model uninterrupted, so that speech features can be applied to the dependency structure analysis model. On the other hand, the model includes the processing of speech and the processing of dependency structure analysis, and the intermediate process does not use gold standard text to process the dependency structure analysis. Since labels may change during the process, there is little research in this area.

[0004] Although existing pipeline-based methods can use off-the-shelf automatic speech recognition tools to transcribe speech into text, there are two important problems: errors in transcription will propagate to subsequent dependency structure analysis; and many acoustic cues that are important for dependency structure analysis (such as rhythm, pauses, etc.) are lost in the transcribed text.

[0005] In end-to-end methods, due to transcription errors, labels may be inconsistent with original labels. Most existing methods achieve this by filtering inconsistent labels, which results in the loss of information that cannot be learned by the model. Summary of the invention

[0006] In order to solve the problem that the existing solutions cannot directly apply speech signals to train dependency structure analysis models, and the technical problem that due to transcription errors, the labels are inconsistent with the original labels, and most of the existing methods are implemented by filtering inconsistent labels, resulting in the loss of these information that cannot be learned by the model, the embodiment of the present invention provides a method and device for dependency structure analysis based on end-to-end speech. The technical solution is as follows:

[0007] On the one hand, a method for end-to-end speech dependency structure analysis is provided, the method being implemented by a device for end-to-end speech dependency structure analysis, the method comprising:

[0008] S1. Acquire speech data, obtain speech feature vectors through a speech feature extractor, input the speech feature vectors into a speech decoder, and obtain speech embedding vector representation;

[0009] S2, mapping the speech embedding vector to the text space based on the adapter to obtain the text embedding vector representation;

[0010] S3, decoding the mapped text representation to obtain a transcribed text, and performing label alignment on the transcribed text to obtain an alignment label of the transcribed text;

[0011] S4. The text is embedded into a vector representation to output the result of speech dependency structure analysis through a dual affine decoder; the speech dependency structure analysis model is trained end-to-end through alignment labels, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

[0012] Optionally, the acquiring of speech data in S1, acquiring a speech feature vector through a speech feature extractor, inputting the speech feature vector into a speech decoder, and obtaining a speech embedding vector representation, includes:

[0013] Get voice data;

[0014] Input the speech data into the speech feature extractor to obtain a speech feature vector representation;

[0015] Get the pre-trained speech decoder;

[0016] The speech feature vector is input into the speech decoder to obtain the speech embedding vector representation.

[0017] Optionally, the adapter in S2 maps the speech embedding vector to the text space to obtain the text embedding vector representation, including:

[0018] Preset adapter; The adapter is implemented by constructing a mapping matrix from the speech decoder vocabulary to the bi-affine decoder vocabulary;

[0019] The speech embedding vector is represented as the corresponding one-hot vector, and the one-hot vector is multiplied with the mapping matrix through matrix multiplication to obtain the text embedding vector representation;

[0020] The speech embedding vector representation is mapped to the text space through the adapter to obtain the text embedding vector representation corresponding to the speech embedding vector representation.

[0021] Optionally, in S3, decoding the mapped text representation to obtain a transcribed text, aligning the transcribed text with a label, and obtaining an aligned label of the transcribed text include:

[0022] Get the gold label text label of the original speech data;

[0023] Align the transcribed text with the gold standard text label to obtain the aligned label of the transcribed text.

[0024] Optionally, in S4, the text is embedded in the vector representation to output the dependency structure analysis result of the speech through a dual affine decoder; the speech dependency structure analysis model is trained end-to-end by aligning the labels, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model, including:

[0025] The text embedding vector representation is input into the dual affine decoder to obtain the dependency relationship between words in the text and output the dependency structure analysis result of the speech;

[0026] The speech dependency structure analysis results and the aligned labels are used to calculate the loss to train the speech dependency structure analysis model end-to-end, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

[0027] On the other hand, a device for analyzing a dependency structure based on end-to-end speech is provided, and the device is applied to a method for analyzing a dependency structure based on end-to-end speech. The device comprises:

[0028] A speech conversion module obtains a speech feature vector through a speech feature extractor, and the speech feature vector is input into a speech decoder to obtain a speech embedding vector representation;

[0029] A data mapping module, used for mapping the speech embedding vector representation to the text space based on the adapter to obtain the text embedding vector representation;

[0030] A label alignment module decodes the text embedding vector representation to obtain a transcribed text, performs label alignment on the transcribed text, and obtains an alignment label of the transcribed text;

[0031] The model training module is used to embed the text into a vector representation and output the result of speech dependency structure analysis through a dual affine decoder; the speech dependency structure analysis model is trained end-to-end through the alignment label, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

[0032] On the other hand, a device for end-to-end speech-based dependency structure analysis is provided, and the device for end-to-end speech-based dependency structure analysis comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for end-to-end speech-based dependency structure analysis is implemented.

[0033] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for end-to-end speech-based dependency structure analysis.

[0034] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0035] In the embodiment of the present invention, an end-to-end approach is adopted to further train the dependency structure analysis model using the feature information in the speech, so as to solve the problem of the separation of the speech and dependency structure analysis tasks in the traditional pipeline method; an alignment tool is used to map the label information of the golden mark text to the transcribed text in the intermediate process, so that even if an error occurs in the transcribed text, complete label information can be obtained. The present invention directly starts with speech to realize the utilization of speech features. In the case where the corresponding label information cannot be obtained due to possible errors in speech transcription, an alignment tool is used to map the label information of the golden mark text to the transcribed text, thereby making full use of speech features and dependency structure analysis data to better train the dependency structure analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a flow chart of a method for end-to-end dependency structure analysis based on speech provided by an embodiment of the present invention;

[0038] Figure 2 It is a framework diagram of a voice end-to-end dependency structure analysis model provided by an embodiment of the present invention;

[0039] Figure 3This is an example diagram of the result of mapping the label of the gold label text to the transcribed text provided by the embodiment of the present invention;

[0040] Figure 4 It is a block diagram of a device for end-to-end speech dependency structure analysis provided by an embodiment of the present invention;

[0041] Figure 5 It is a structural diagram of a device for end-to-end speech dependency structure analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0043] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0044] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0045] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0046] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0047] The embodiment of the present invention provides a method for end-to-end voice dependency structure analysis, which can be implemented by an end-to-end voice dependency structure analysis device, which can be a terminal or a server. Figure 1 The flowchart of the method for end-to-end dependency structure analysis based on speech is shown in FIG. 1 . The processing flow of the method may include the following steps:

[0048] S1. Acquire voice data, perform embedded conversion on the voice data, and obtain voice embedded data;

[0049] In a feasible implementation, the acquisition of speech data in S1, the acquisition of speech feature vectors by a speech feature extractor, the input of the speech feature vectors into a speech decoder, and the acquisition of speech embedding vector representation include:

[0050] Get voice data;

[0051] Inputting the speech data into a speech feature extractor to obtain a speech feature vector representation;

[0052] Get the pre-trained speech decoder;

[0053] The speech feature vector is input into the speech decoder to obtain a speech embedding vector representation.

[0054] In a feasible implementation manner, the model framework of the speech end-to-end dependency structure analysis method of the present invention includes four parts: a speech feature extractor, a speech decoder, an adapter and a dual affine decoder, as shown in FIG. Figure 2 shown.

[0055] In a feasible implementation, a speech feature extractor extracts speech features from the original speech input to obtain a speech feature vector, which is then input into a speech decoder to obtain a speech embedding vector representation. In order to improve the performance of the task, a pre-trained speech decoder, such as Whisper, can be used. These models can convert speech feature vectors into speech embedding vector representations.

[0056] Speech data is a sound signal, and a speech feature vector refers to some features related to speech, such as pitch, frequency, etc. The speech embedding vector representation is a vector input into the decoder to obtain a high-dimensional representation of speech.

[0057] S2, mapping the speech embedding vector representation to the text space based on the adapter to obtain the output text embedding vector representation;

[0058] In a feasible implementation, the adapter-based mapping of the speech embedding vector representation to the text space in S2 to obtain the text embedding vector representation includes:

[0059] Preset adapter;

[0060] The adapter is implemented by constructing a mapping matrix from the speech decoder vocabulary to the bi-affine decoder vocabulary;

[0061] The speech embedding vector is represented as the corresponding one-hot vector, and the one-hot vector is multiplied with the mapping matrix through matrix multiplication to obtain the text embedding vector representation;

[0062] The speech embedding data is mapped to the text space through the adapter to obtain the output text embedding vector representation corresponding to the speech embedding data.

[0063] In one possible implementation, it is not feasible to directly use speech embeddings for text-related tasks because of the difference in representation space between speech and text. To address this issue, we introduce an adapter whose role is to map speech embeddings into the text space, where the text space is the space of the bi-affine decoder used later. The adapter is implemented by constructing a mapping matrix from the speech decoder vocabulary to the bi-affine decoder vocabulary. First, the speech embeddings are converted to the corresponding one-hot vectors, which are then multiplied with the mapping matrix through matrix multiplication to obtain the text embedding vector representation.

[0064] S3, decoding the text embedding vector representation to obtain a transcribed text, and performing label alignment on the transcribed text to obtain an alignment label of the transcribed text;

[0065] In a feasible implementation manner, decoding the mapped text representation to obtain a transcribed text in S3, and performing label alignment on the transcribed text to obtain an alignment label of the transcribed text include:

[0066] Get the gold label text label of the original speech data;

[0067] Align the transcribed text with the golden label text label to obtain the aligned label of the transcribed text;

[0068] S4. The text is embedded in the vector representation and the dependency structure analysis result of the speech is outputted through a dual affine decoder. The steps described above do not cause gradient interruption. The speech dependency structure analysis model is trained end-to-end through the speech dependency structure analysis result. The speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

[0069] In a feasible implementation, the text embedding vector representation in S4 outputs the dependency structure analysis result of the speech through the dual affine decoder, including:

[0070] Inputting the text embedding vector representation into a dual affine decoder to obtain the dependency relationship between words in the text, and outputting the dependency structure analysis result of the speech;

[0071] None of the steps causes gradient interruption, and the speech dependency structure analysis results and the aligned labels are used to calculate the loss to train the speech dependency structure analysis model end-to-end, and the speech dependency structure analysis is completed based on the speech dependency structure analysis model after end-to-end training.

[0072] In a feasible implementation, the bi-affine decoder predicts whether there is a dependency relationship between two words and its dependency label through a bi-affine classifier. The bi-affine classifier can effectively capture the semantic and syntactic relationships between words, providing a basis for dependency parsing.

[0073] In a feasible implementation, during the above process, the transcribed text of the text embedding representation obtained by the speech decoder may be inconsistent with its true text. However, the labels of the dependency structure analysis only apply to the true text. Therefore, an alignment tool is needed to map the label information of the true text into the transcribed text.

[0074] In a feasible implementation, this application uses an alignment model to achieve label alignment. Based on the source language sentences and target language sentences in the parallel corpus, a neural network model is established, and the model learns the alignment through multiple iterations. As Figure 3 shown, it shows how to obtain the label information of the dependency structure analysis corresponding to the transcribed text when the transcribed text is inconsistent with the gold standard text. When there are word transcription errors, such as transcription errors caused by similar pronunciations or differences in number case (e.g., uppercase and lowercase) or traditional and simplified Chinese characters, but after using the alignment tool, the mapping can be done normally. For example, "他" is transcribed as "她", "二" is transcribed as "2", and "夺冠" is transcribed as "奪冠". If the father nodes of these words are transcribed correctly or have the same transcription error, then these words will directly inherit the label information of the original gold standard text. For example, the label of "奪冠" is still "HED", and the root node is its father node. However, it is possible that information in the gold standard text is missing after transcription. For example, in the example, the word "了" is missing, and there is no word with "了" as its father node, so no modification is needed. In addition, there may be other words in the transcribed text that do not appear in the gold standard text. Since we do not know the label information of this word, we will point the word pointed to by the root node to this word, and its label is set to "none". In this way, even if the transcribed text and the gold standard text are inconsistent, through mapping and corresponding processing measures, we can obtain the label information corresponding to the transcribed text, thus better helping the dependency structure analysis model for training.

[0075] In the embodiments of the present invention, the main objectives are the problems of error propagation, inability to utilize acoustic features, and mapping labels in dependency structure analysis under speech. Here, we propose an end-to-end dependency structure analysis method that directly starts from speech, realizes the utilization of speech features, and when the corresponding label information cannot be obtained due to possible errors in speech encoding, uses an alignment tool to map the label information of the gold standard text to the transcribed text, so as to make full use of the data of dependency structure analysis to better train the dependency structure analysis model.

[0076] Figure 4The present invention is a block diagram of a device for end-to-end speech dependency structure analysis according to an exemplary embodiment. The device is used in a method for end-to-end speech dependency structure analysis. Figure 4 The device includes a speech conversion module 310, a data mapping module 320, a transcription module 330 and a label alignment module 340. Among them:

[0077] The speech conversion module 310 is used to obtain speech data, obtain speech feature vectors through a speech feature extractor, and input the speech feature vectors into a speech decoder to obtain a speech embedding vector representation;

[0078] A data mapping module 320, configured to map the speech embedding vector representation to the text space based on the adapter to obtain the text embedding vector representation;

[0079] A label alignment module 330 is used to decode the text embedding vector representation to obtain a transcribed text, perform label alignment on the transcribed text, and obtain an alignment label of the transcribed text;

[0080] The model training module 340 is used to embed the text into a vector representation and output the result of the speech dependency structure analysis through a dual affine decoder; none of the steps causes the gradient to be interrupted, and the speech dependency structure analysis model is trained end-to-end through the alignment labels, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

[0081] Optionally, the voice conversion module 310 is used to obtain voice data;

[0082] Inputting the speech data into a speech feature extractor to obtain a speech feature vector representation;

[0083] Get the pre-trained speech decoder;

[0084] The speech feature vector is input into the speech decoder to obtain a speech embedding vector representation.

[0085] Optionally, the data mapping module 320 is used to preset the adapter;

[0086] Preset adapter;

[0087] The speech embedding vector representation is mapped to the text space through an adapter to obtain the text embedding vector representation corresponding to the speech embedding vector representation.

[0088] Optionally, mapping the speech embedding vector representation to the text space through an adapter to obtain the text embedding vector representation corresponding to the speech embedding vector representation includes:

[0089] The adapter is implemented by constructing a mapping matrix from the speech decoder vocabulary to the bi-affine decoder vocabulary;

[0090] The speech embedding vector is represented as the corresponding one-hot vector, and the one-hot vector is multiplied with the mapping matrix through matrix multiplication to obtain the text embedding vector representation.

[0091] Optionally, a transcription module 330 for obtaining a gold-labeled text label of the original speech data;

[0092] Align the transcribed text with the gold standard text label to obtain the aligned label of the transcribed text.

[0093] Optionally, a label alignment module 340 is used to input the text embedding vector representation into a dual affine decoder to obtain the dependency relationship between words in the text and output the dependency structure analysis result of the speech;

[0094] None of the steps causes gradient interruption, and the speech dependency structure analysis results and the aligned labels are used to calculate the loss to train the speech dependency structure analysis model end-to-end, and the speech dependency structure analysis is completed based on the trained speech end-to-end model.

[0095] In the embodiment of the present invention, the main goal is to build a speech dependency structure analysis model that can be learned end-to-end. Due to the existence of error propagation and inability to utilize acoustic features in the pipeline method, as well as the problem of mapping labels when using speech signals to train the dependency structure analysis model, we propose an end-to-end dependency structure analysis method here, starting directly from speech, to realize the utilization of speech features, and in the case where the corresponding label information cannot be obtained due to possible errors in speech encoding, an alignment tool is used to map the label information of the gold label text to the transcribed text, thereby making full use of the dependency structure analysis data to better train the speech dependency structure analysis model.

[0096] Figure 5 is a schematic diagram of a structure of a device for end-to-end speech dependency structure analysis provided by an embodiment of the present invention, such as Figure 5 As shown, the end-to-end voice dependency structure analysis device may include the above Figure 4 Optionally, the device 410 for analyzing the dependency structure based on end-to-end speech may include a first processor 2001 .

[0097] Optionally, the speech-based end-to-end dependency structure analysis device 410 may further include a memory 2002 and a transceiver 2003 .

[0098] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0099] Combine the following Figure 5The components of the end-to-end voice-based dependency structure analysis device 410 are specifically introduced as follows:

[0100] The first processor 2001 is the control center of the end-to-end voice dependency structure analysis device 410, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0101] Optionally, the first processor 2001 can perform various functions of the speech-based end-to-end dependency structure analysis device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0102] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0103] In a specific implementation, as an embodiment, the speech end-to-end dependency structure analysis device 410 may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0104] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0105] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0106] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0107] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0108] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and is connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0109] It should be noted that Figure 5 The structure of the end-to-end voice-based dependency structure analysis device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0110] In addition, the technical effects of the end-to-end voice dependency structure analysis device 410 can refer to the technical effects of the end-to-end voice dependency structure analysis method described in the above method embodiment, and will not be repeated here.

[0111] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0112] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0113] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0114] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0115] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0116] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0123] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for end-to-end dependency structure analysis based on speech, characterized in that: The method comprises: S1, obtaining speech data, obtaining a speech feature vector through a speech feature extractor, and inputting the speech feature vector into a speech decoder to obtain a speech embedding vector representation; S2, mapping the speech embedding vector to the text space based on the adapter to obtain the text embedding vector representation; S3, decoding the mapped text representation to obtain a transcribed text, and performing label alignment on the transcribed text to obtain an alignment label of the transcribed text; S4. Embed the text into a vector representation and output the result of speech dependency structure analysis through a dual affine decoder; train the speech dependency structure analysis model end-to-end through the alignment labels, and complete the speech dependency structure analysis based on the trained speech dependency structure analysis model.

2. The method for end-to-end speech dependency structure analysis according to claim 1, characterized in that: The step of obtaining speech data in S1, obtaining a speech feature vector through a speech feature extractor, inputting the speech feature vector into a speech decoder, and obtaining a speech embedding vector representation includes: Get voice data; Inputting the speech data into a speech feature extractor to obtain a speech feature vector representation; Get the pre-trained speech decoder; The speech feature vector is input into the speech decoder to obtain a speech embedding vector representation.

3. The method for end-to-end speech dependency structure analysis according to claim 2, characterized in that: The adapter-based mapping of the speech embedding vector to the text space in S2 to obtain the text embedding vector representation includes: Preset adapter; the adapter is implemented by constructing a mapping matrix from a speech decoder vocabulary to a dual affine decoder vocabulary; The speech embedding vector is represented as the corresponding one-hot vector, and the one-hot vector is multiplied with the mapping matrix through matrix multiplication to obtain the text embedding vector representation; The adapter maps the speech embedding vector representation to the text space to obtain the text embedding vector representation corresponding to the speech embedding vector representation.

4. The method for end-to-end speech dependency structure analysis according to claim 3, characterized in that: In S3, decoding the mapped text representation to obtain a transcribed text, and performing label alignment on the transcribed text to obtain an aligned label of the transcribed text includes: Get the gold label text label of the original speech data; The transcribed text is label-aligned with the gold-labeled text label to obtain an alignment label of the transcribed text.

5. The method for end-to-end speech dependency structure analysis according to claim 4, characterized in that: The text embedding vector representation in S4 outputs the dependency structure analysis result of the speech through the dual affine decoder; The speech dependency structure analysis model is trained end-to-end by the alignment label, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model, including: Inputting the text embedding vector representation into a dual affine decoder to obtain the dependency relationship between words in the text, and outputting the dependency structure analysis result of the speech; The speech dependency structure analysis result and the aligned labels are used to calculate the loss to train the speech dependency structure analysis model end-to-end, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

6. A device for end-to-end speech dependency structure analysis, the device for end-to-end speech dependency structure analysis is used to implement the method for end-to-end speech dependency structure analysis as claimed in any one of claims 1 to 5, characterized in that: The device comprises: A speech conversion module is used to obtain speech data, obtain speech feature vectors from the speech data through a speech feature extractor, and input the speech feature vectors into a speech decoder to obtain a speech embedding vector representation; A data mapping module, used for mapping the speech embedding vector representation to the text space based on the adapter to obtain the text embedding vector representation; A label alignment module decodes the mapped text embedding vector representation to obtain a transcribed text, performs label alignment on the transcribed text, and obtains an alignment label of the transcribed text; The model training module is used to embed the text into a vector representation and output the result of speech dependency structure analysis through a dual affine decoder; the speech dependency structure analysis model is trained end-to-end through the alignment label, and the speech dependency structure analysis is completed based on the trained speech dependency structure analysis model.

7. A device for end-to-end speech dependency structure analysis, characterized in that: The end-to-end voice-based dependency structure analysis device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images