Text conversion method, training method of text conversion model and electronic equipment

Abstract

The invention provides a text conversion method, a text conversion model training method and electronic equipment, and the method comprises the steps: obtaining pre-training data, carrying out the pre-training of a first large model based on the pre-training data, and obtaining a second large model; fine-tuning training data are obtained, fine-tuning training is conducted on the second large model based on the fine-tuning training data, a text conversion model is obtained, and the fine-tuning training data comprise first training data obtained by converting the spoken language text into the written language text and second training data obtained by converting the written language text into the spoken language text. According to the training method, a model is trained through first training data obtained by converting spoken language texts into written language texts and second training data obtained by converting the written language texts into the spoken language texts at the same time, and a text conversion model capable of converting the spoken language texts into the written language texts and converting the written language texts into the spoken language texts can be obtained.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a text conversion method, a training method for the conversion model, and an electronic device. Background Technology

[0002] In daily meetings or other communications, it is usually necessary to convert spoken text into written form for easier reading, such as organizing the spoken content of each person's speech into written text. In some intelligent scenarios, such as digital human live streaming and robot telemarketing, it is necessary to convert the written text generated by the intelligent model into spoken text to make it more natural.

[0003] However, current text conversion models can only perform the task of converting spoken text into written text, or only the task of converting written text into spoken text. In other words, current text conversion models cannot meet the need to handle both conversions simultaneously. Summary of the Invention

[0004] This application provides a text conversion method and a training method and electronic device for the conversion model. The model is trained simultaneously using first training data for converting spoken text into written text and second training data for converting written text into spoken text, thereby obtaining a text conversion model that can convert both spoken and written text into spoken text.

[0005] In a first aspect, this application provides a training method for a text conversion model. The training method for a text conversion model includes: acquiring pre-training data, pre-training a first large model based on the pre-training data to obtain a second large model; acquiring fine-tuning training data, fine-tuning the second large model based on the fine-tuning training data to obtain the text conversion model, wherein the fine-tuning training data includes first training data for converting spoken text into written text and second training data for converting written text into spoken text.

[0006] In this application, a large amount of unlabeled data is used to perform unsupervised pre-training on the first large model, giving full play to the generalization ability of the large model. Then, the second large model is fine-tuned and trained using instruction data for converting spoken text into written text and instruction data for converting written text into spoken text, to obtain a text conversion model. That is, the large model is targeted for fine-tuning and training using instruction fine-tuning data. Under this training method, the obtained text conversion model can learn the conversion between spoken and written language and between spoken and written language, and can simultaneously have the function of bidirectional text style conversion between spoken and written text and between written and spoken text.

[0007] In some embodiments, obtaining pre-training data includes: obtaining the text of instruction data to be generated; constructing an input template for generating instruction-type data; obtaining a first instruction data pair set based on the text of the instruction data to be generated and the input template; evaluating and filtering multiple instruction data pairs in the first instruction data pair set to obtain a filtered first instruction data pair set; obtaining a second instruction data pair set and unlabeled data; and obtaining the pre-training data based on the filtered first instruction data pair set, the second instruction data pair set, and the unlabeled data.

[0008] In this embodiment, by utilizing large commercial models to generate and evaluate data, and removing low-scoring instruction data, a comprehensive high-quality instruction dataset is obtained, which improves the efficiency of generating high-quality instruction data. Furthermore, a large amount of high-quality instruction data can be added to the pre-training data to enhance the model's performance in zero-shot and few-shot scenarios, thus helping to improve the model's bidirectional text conversion capabilities.

[0009] In some embodiments, a method for obtaining the first training data and the second training data includes: obtaining a text-to-text conversion data pair set, wherein the text-to-text conversion data pair set includes a plurality of spoken texts and written texts corresponding to the plurality of spoken texts; constructing the first training data based on the text-to-text conversion data pair set and instructions for converting spoken texts into written texts; and constructing the second training data based on the text-to-text conversion data pair set and instructions for converting written texts into spoken texts.

[0010] In this embodiment, different training data are constructed by adding different conversion instructions to the text conversion data pair set. This allows the first training data and the second training data to share the same text conversion data pair set. Furthermore, by adding different conversion instructions to construct the training data, it is convenient to train the large model to learn the conversion rules between spoken and written text, thereby improving its ability to convert spoken text into written text and vice versa.

[0011] In some embodiments, the first large model includes an expanded vocabulary for segmenting the input text, and the training method further includes: acquiring a segmentation training corpus; training the segmentation model using the segmentation training corpus to obtain the expanded vocabulary; acquiring an initial vocabulary of the first large model, and merging the expanded vocabulary and the initial vocabulary to obtain the merged vocabulary.

[0012] In this embodiment, by expanding the original vocabulary of the first large model, subsequent large models perform word segmentation on the input text based on this merged vocabulary. This expands the Chinese characters in the vocabulary, improves the efficiency of encoding and decoding, and can segment words that appear too infrequently in the training corpus, thereby improving the accuracy of model generation.

[0013] In some embodiments, the first large model includes an encoding module, and the training method includes: when pre-training the first large model, adding bias training parameters to the query matrix, key matrix, and value matrix in the encoding module for pre-training.

[0014] In this embodiment, the extrapolation capability of the text conversion model can be improved by adding a bias term during pre-training.

[0015] In some embodiments, the second large model includes a first mapping layer for mapping the input text after word segmentation, and the training method further includes: when fine-tuning the second large model, adding noise to the output of the first mapping layer and then fine-tuning the training.

[0016] In this embodiment, adding noise to the output of the first mapping layer of the second model before outputting it can improve the effect of instruction fine-tuning and improve the effect of the text conversion model when converting text.

[0017] In some embodiments, the second large model further includes a second mapping layer for mapping the positions of the segmented input text, and the training method further includes: when fine-tuning the second large model, if the positions of the segmented input text exceed the total number of position codes of the second mapping layer, then a linear interpolation method is used to increase the total number of position codes.

[0018] In this embodiment, by using linear interpolation to increase the total number of position codes, the model can be adapted to a longer context window.

[0019] Secondly, this application provides a text conversion method, which includes: obtaining a first text to be converted and a conversion instruction; inputting the first text to be converted and the conversion instruction into a text conversion model to obtain a first converted text, wherein the text conversion model is trained using the training method described in any embodiment of the first aspect.

[0020] In this embodiment, the text conversion model trained by the training method described in any one of the embodiments of the first aspect is used to convert the first text to be converted, thereby achieving the purpose of converting spoken text into written text and written text into spoken text.

[0021] In some embodiments, inputting the first text to be converted and the conversion instruction into a text conversion model to obtain the first converted text includes: determining whether different identity information exists based on the first text to be converted; if different identity information exists, segmenting the first text to be converted according to the different identity information to obtain multiple second texts to be converted; if no different identity information exists, segmenting the first text to be converted to obtain multiple second texts to be converted; or, not processing the first text to be converted, using the first text to be converted as the second text to be converted; and inputting the second text to be converted and the conversion instruction into the text conversion model to obtain the first converted text.

[0022] In this embodiment, the first text to be converted is segmented based on identity information or segmented based on preset rules, which can reduce the processing pressure on the text conversion model.

[0023] In some embodiments, the step of inputting the second text to be converted and the conversion instruction into the text conversion model to obtain the first converted text includes: if the conversion instruction is an instruction to convert written text into spoken text, then inputting the second text to be converted and the conversion instruction into the text conversion model to obtain the second converted text; if the conversion instruction is an instruction to convert spoken text into written text, then determining whether the second text to be converted needs to be filtered, and if so, using empty text as the second converted text, and if not, inputting the second text to be converted and the conversion instruction into the text conversion model to obtain the second converted text; determining whether the second converted text needs to be merged, and if so, obtaining the second converted text to be merged and merging the second converted text to be merged to obtain the first converted text, and if not, using the second converted text as the first converted text.

[0024] In this embodiment, when converting spoken text into written text, the text to be converted is pre-determined to determine whether filtering is required. If so, it is filtered, and the first converted text is obtained after merging. This reduces the processing pressure on the text conversion model.

[0025] In some embodiments, the method further includes: simultaneously displaying the first text to be converted and the first converted text on a display interface; and / or, displaying the first text to be converted on the display interface, and displaying added or deleted text annotations or identity information on the first text to be converted based on the first converted text.

[0026] In this embodiment, visualizing the text before and after conversion helps users better understand and process the results of the text conversion.

[0027] Thirdly, embodiments of this application provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in either the first or second aspect.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method described in either the first or second aspect.

[0029] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the first or second aspect above. Attached Figure Description

[0030] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise stated, the figures in the drawings do not constitute a limitation on scale.

[0031] Figure 1 This is a schematic diagram of a text conversion application scenario provided in an embodiment of this application;

[0032] Figure 2 This is a structural block diagram of an electronic device provided in an embodiment of this application;

[0033] Figure 3 This is a flowchart illustrating a training method for a text conversion model provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the structure of a first major model provided in an embodiment of this application;

[0035] Figure 5 This is a flowchart illustrating a text conversion method provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of a display interface provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of another display interface provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of another display interface provided in the embodiments of this application. Detailed Implementation

[0039] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0040] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0041] It should be noted that, unless there is a conflict, the various features in the embodiments of this invention can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram, in some cases, they can be divided differently from those in the device. In addition, the terms "first," "second," etc., used herein do not limit the data or execution order, but only distinguish identical or similar items with substantially the same function and effect.

[0042] In everyday meetings and other discussions, it's often necessary to convert spoken language into written form for easier reading, such as transcribing spoken text into written text. Conversely, in intelligent scenarios, such as virtual hosts or automated customer service calls, it's crucial to convert written text generated by intelligent models into spoken text to make the converted speech more natural. However, current text conversion models can only handle either spoken to written or written to spoken conversion, not both simultaneously. Furthermore, due to a lack of sufficient training data, these models don't perform ideally when performing conversions.

[0043] To address the aforementioned technical problems, embodiments of this application provide a text conversion method and a training method and electronic device for the conversion model. In the training method, the model is trained simultaneously using first training data for converting spoken text to written text and second training data for converting written text to spoken text, thereby obtaining a text conversion model that can convert both spoken and written text into spoken text.

[0044] To facilitate understanding of this application, the following describes exemplary applications of the electronic devices provided in the embodiments of this application for training text conversion models or for text conversion. It is understood that the electronic devices can both train the text conversion model and use the text conversion model to perform text conversion.

[0045] The electronic devices provided in some embodiments of this application can be various types of terminals such as laptops, desktop computers, or mobile devices. When the terminal is used to train a text conversion model, those skilled in the art input a prepared training set into the terminal, design a large model on the terminal, or use an open-source large model. The terminal uses the training set to iteratively train the large model, determines the final model parameters, and then configures the large model with these final model parameters to obtain the text conversion model. When the terminal is used for text conversion, it calls the built-in text conversion model to perform corresponding calculations on the first text to be converted provided by other devices or the user, generating the corresponding first converted text.

[0046] As an example, see Figure 1 , Figure 1 This is a schematic diagram of an application scenario for the text conversion method provided in this application embodiment. The terminal 10 connects to the server 20 through a network, where the network can be a wide area network or a local area network, or a combination of both.

[0047] Terminal 10 can be used to acquire training sets and build large models. For example, those skilled in the art can download prepared training sets on the terminal and build the network structure of a large model. It is understood that terminal 10 can also be used to acquire text to be converted. For example, a user inputs text to be converted through an input interface, and after the input is completed, the terminal automatically acquires the text to be converted.

[0048] In some embodiments, terminal 10 locally executes the method for training a text conversion model provided in this application to train a large model using a training set, determine the final model parameters, and then configure the large model with the final model parameters to obtain a text conversion model. In some embodiments, terminal 10 may also send a training set and a constructed large model stored on the terminal by a person skilled in the art to server 20 via a network. Server 20 receives the training set and the large model, trains the designed large model using the training set, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters so that the large model can be configured with the final model parameters to obtain a text conversion model.

[0049] In some embodiments, terminal 10 locally executes the text conversion method provided in this application embodiment to provide text conversion services to the user, calls the built-in text conversion model, performs corresponding calculations on the text to be converted input by the user, and generates converted text. In some embodiments, terminal 10 can also send the text to be converted input by the user on the terminal to server 20 via the network. Server 20 receives the text to be converted, calls the built-in text conversion model to perform corresponding calculations on the text to be converted, generates converted text, and then sends the converted text to terminal 10. After receiving the converted text, terminal 10 can display the converted text on its own display interface for the user to view.

[0050] The structure of the electronic device in the embodiments of this application is described below. Figure 2 This is a schematic diagram of the structure of an electronic device 100 in an embodiment of this application. The electronic device 100 includes at least one processor 110, a memory 150, at least one network interface 120, and a user interface 130. The various components in the electronic device 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 140.

[0051] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0052] User interface 130 includes one or more output devices 131 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 130 also includes one or more input devices 132, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0053] Memory 150 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory. Memory 150 may optionally include one or more storage devices physically located away from processor 110.

[0054] In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0055] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0056] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 including Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0057] Display module 153 is configured to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 131 (e.g., display screen, speaker, etc.) associated with user interface 130.

[0058] The input processing module 154 is used to detect and translate one or more user inputs or interactions from one or more input devices 132.

[0059] As can be understood from the above, the method for training a text conversion model and the text conversion method provided in the embodiments of this application can be implemented by various types of electronic devices with computing capabilities, such as smart terminals and servers.

[0060] The following describes the method for training a text conversion model provided in this application, using exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 3 , Figure 3 This is a flowchart illustrating a method for training a text conversion model according to an embodiment of this application. This training method can be applied to electronic devices, and the method includes:

[0061] Step S10: Obtain pre-training data, and pre-train the first large model based on the pre-training data to obtain the second large model.

[0062] Pre-training data can be a large amount of general, unlabeled text training data. Electronic devices can obtain this data from large-scale text corpora, such as Wikipedia, news articles, online text, books, forum posts, etc., or from web page text data obtained through web crawlers. Pre-training data typically contains various language expressions and contexts, which helps large models learn a wider range of language knowledge.

[0063] The first major model can be an open-source large language model (LLM), such as LLaMA 2, Baichuan, etc., or it can be a self-built large model.

[0064] In some implementations, the structure of the first major model can be as follows: Figure 4 As shown, it includes a mapping module 210, an encoding module 220, and a classification module 230 connected in sequence. The mapping module 210 includes a first mapping layer 211 for mapping the segmented input text and a second mapping layer 212 for mapping the position of the segmented input text. Specifically, the first mapping layer 211 includes a token embedding layer, which converts the segmented input text into fixed-dimensional word vectors. The segmented input text can be a sequence of basic units (token sequences) that can be processed by the first main model. The second mapping layer 212 includes a position embedding layer, which converts the position information of the segmented input text into positional encoding. The first main model can capture the order information of the segmented input text in the sequence based on this positional encoding. The encoding module 220 may include multiple transformer decoder layers connected in sequence. The encoding module 220 can convert the output of the mapping module 210 into the corresponding token encoding. The classification module 230 can use the softmax function to convert the output of the encoding module 220 into a probability distribution. It can be understood that the output of the encoding module 220 should be the output of the last transformer decoder layer.

[0065] exist Figure 4In the first major model of the structure shown, it can predict the probability of the next basic unit sequence (token sequence) appearing based on an input basic unit sequence (token sequence). This is done by repeating the word chain until the generated basic unit sequence (token sequence) reaches a certain length or meets other loop termination conditions, thus obtaining a complete string composed of the generated basic unit sequences (token sequences).

[0066] Specifically, the string processing flow can include the following steps: Step 1: Input the initial text or instruction that starts the first main model to generate text. This initial text or instruction serves as the starting point of the generation process and can be a paragraph, sentence, phrase, or simply a word; Step 2: The first main model segments the input initial text or instruction into a sequence of basic units (i.e., a token sequence), such as using a Byte Pair Encoding (BPE) algorithm or a word-level (WordPiece) algorithm to segment the initial text or instruction into multiple token sequences: t1, t2, t3...t k Step 3: Convert each token sequence into word vectors through the first mapping layer 211 (e.g., token embedding layer); Step 4: Convert each token sequence into corresponding positional codes through the second mapping layer 212 (e.g., position embedding layer); Step 5: Input each token sequence into the encoding module 220 for encoding; Step 6: Pass the output of the last transformer decoder layer in the encoding module 220 to the classification module 230, which will use the softmax function to calculate the probability distribution of each token in the vocabulary; Step 7: Generate the next token sequence t based on the selection strategy and the probability distribution output by the classification module 230. k+1 ', and generate the next token sequence t k+1 'Add to the input token sequence, where the selection strategy can be greedy sampling, beam search, etc. Step 8: Re-input the updated token sequence into the first main model, and repeat steps 3 to 8 until the generated length reaches the predetermined length or other loop termination conditions are met, resulting in a sequence consisting of t1', t2', t3'...t k '、t k+1 '、……t n A complete string composed of '.

[0067] The work of each layer in the first major model can be described by the mathematical expression y = a(W·x + b). From a physical perspective, the work of each layer in the first major model can be understood as completing the transformation from the input space to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimensional increase / decrease; 2. Magnification / reduction; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by "W·x", operation 4 is completed by "+b", and operation 5 is implemented by "a()". The reason for using the word "space" is that the object being classified is not a single thing, but a class of things. Space refers to the set of all individuals of this class of things. Here, W is the weight matrix of each layer of the first major model. Each value in this matrix represents the weight value of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space mentioned above. That is, the matrix W of each layer of the first major model controls how the space is transformed. The pre-training process of the first major model is essentially about learning how to transform the control space, and more specifically, learning the weight matrix.

[0068] In this step, the first large model is iteratively trained using pre-trained data. That is, the pre-set weight matrix of the first large model is iteratively trained using pre-trained data until the first large model converges. At this point, the weight matrix of all layers is the weight matrix of all layers after pre-training. Finally, the first large model is configured with this final weight matrix, which is the second large model.

[0069] Step S20: Obtain fine-tuning training data, and fine-tune the second model based on the fine-tuning training data to obtain the text conversion model. The fine-tuning training data includes the first training data for converting spoken text into written text and the second training data for converting written text into spoken text.

[0070] Spoken texts refer to texts written in the language used during everyday oral conversations. These texts typically possess characteristics of spoken language, such as its discreteness, spontaneity, and non-standardization. Spoken texts can include daily dialogues, speeches, and oral meeting minutes. Their language is closer to everyday spoken language and has colloquial features, including common phrases, catchphrases, and accent characteristics.

[0071] Written texts refer to the texts written and read using the language used in writing and reading. Their language is more standardized and formal, and they typically exhibit normativity, rigor, and standardization. This form of text is commonly used in written documents, reports, papers, and literary works, and usually includes standard grammatical structures, formal vocabulary selection, and adherence to the norms of written language expression.

[0072] The first training data, i.e., the instruction data for converting spoken text into written text, includes the spoken text, the corresponding written text, and the instructions for converting spoken text into written text. The second training data, i.e., the instruction data for converting written text into spoken text, includes the written text, the corresponding spoken text, and the instructions for converting written text into spoken text. Therefore, the first and second training data differ only in the conversion instructions; thus, both the spoken and written texts can be used interchangeably in both training data.

[0073] Similarly, the fine-tuning training process of the second major model is essentially about learning how to control the transformation space, more specifically, learning the weight matrices of each layer. In this step, the second major model is iteratively trained using fine-tuning training data, that is, the weight matrices configured for the second major model are iteratively trained using the fine-tuning training data until the second major model converges. At this point, the weight matrices of all layers are the weight matrices of all layers after fine-tuning training. Finally, the second major model is configured with this final weight matrix, which is the text conversion model.

[0074] By using the first and second training data as task-based data for bidirectional conversion between spoken and written texts, the second model can be fine-tuned and trained. This allows the second model to better learn the linguistic features between spoken and written texts, understand the differences between spoken and written language, and improve its ability to convert spoken texts into written texts and vice versa, thus achieving bidirectional conversion.

[0075] In summary, this application firstly pre-trains the first large model using conventional general training corpora to obtain the second large model. This involves pre-training the large model with a large amount of unsupervised data (such as unlabeled data) to fully leverage its generalization ability and improve its applicability and effectiveness across various related tasks. This allows the large model to learn broader and more general patterns and rules, resulting in better performance across various downstream tasks. Next, the second large model is fine-tuned using instruction data for converting spoken text to written text, and vice versa, to obtain the text conversion model. This involves targeted fine-tuning of the large model using instruction fine-tuning data. Under this training method, the resulting text conversion model can simultaneously learn spoken-to-written and written-to-spoken conversion, possessing bidirectional text style conversion capabilities for both spoken-to-written and written-to-spoken text conversion.

[0076] To further improve the bidirectional text style conversion performance of the trained text conversion model, in some embodiments, pre-training data is obtained, including:

[0077] Step S11: Obtain the text of the instruction data to be generated.

[0078] The text to be generated as instruction data can be general public text data. Electronic devices can obtain it from large-scale text corpora, such as Wikipedia, news articles, online text, books, forum posts, etc., or they can obtain web page text data through web crawlers.

[0079] Step S12: Construct an input template for generating instruction-type data.

[0080] This input template serves as a prompt for generating instruction-type data. It instructs the generative model to generate various task-type data pairs from the text containing the instruction data, such as translation task pairs, question-answering task pairs, and text conversion task pairs. Understandably, not all text can generate suitable instructions; therefore, when constructing the input template, the generative model will be allowed to generate "None" instructions for text that cannot produce suitable instructions.

[0081] Step S13: Based on the text of the instruction data to be generated and the input template, obtain the first instruction data pair set.

[0082] Next, the text of the instruction data to be generated and the input template are input into the generative model. The generative model generates data, and after filtering the generated data for None instructions and their corresponding texts, a first set of instruction data pairs is obtained. The generative model can be a suitable model such as ChatGPT. The first set of instruction data pairs includes data pairs from various tasks, such as translation data pairs for translation tasks, question-and-answer data pairs for question-and-answer tasks, and text-to-text conversion data pairs for text-to-text conversion tasks.

[0083] Step S14: Evaluate and filter multiple instruction data pairs in the first instruction data pair set to obtain the filtered first instruction data pair set.

[0084] Specifically, an input template for evaluation instruction data can be constructed. This template, along with a set of first instruction data pairs, is then input into multiple different commercial models. These models evaluate and score multiple instruction data pairs within the generated set of first instruction data pairs. Based on the scores from these models, the data is filtered. For example, instruction data pairs with consistent scores and an average score exceeding a certain value are retained, while data pairs with inconsistent scores or an average score below a certain value are discarded. This results in a filtered set of first instruction data pairs. The commercial models can employ suitable models such as Wenxin Yiyan or Zhisi Tianxia to score the instruction data pairs.

[0085] Step S15: Obtain the second instruction data pair set and unlabeled data.

[0086] The second set of instruction data pairs can be an internally annotated or open-source set of instruction data pairs. It can include data pairs from various tasks, such as translation data pairs for translation tasks, question-and-answer data pairs for question-and-answer tasks, and text conversion data pairs for text conversion tasks.

[0087] Unlabeled data can be general public text data, such as data obtained from Wikipedia, news articles, online text, books, forum posts, etc., or data obtained from public datasets.

[0088] Step S16: Based on the filtered first instruction data pair set, second instruction data pair set, and unlabeled data, obtain pre-training data.

[0089] Specifically, the first instruction data pair set, the second instruction data pair set, and the unlabeled data are fused in a certain proportion to obtain pre-training data. The specific fusion ratio can be set according to actual needs and is not limited here.

[0090] It is understandable that multi-task instructions can enhance the zero-shot and few-shot performance of pre-trained models. However, the amount of internally labeled data or open-source instruction data is far less than the amount of unlabeled data. If the first model is pre-trained using only internally labeled instruction data, open-source instruction data, and unlabeled data, the zero-shot and few-shot performance it brings is almost forgotten in the massive training on unlabeled data.

[0091] In this embodiment, by utilizing commercial large-scale model generation and evaluation data, and removing low-scoring instruction data, a comprehensive high-quality instruction dataset is obtained. Compared to using expensive manually labeled data or open-source instruction data, this approach improves the quality of instruction data and rapidly generates a large number of high-quality instruction datasets, thereby increasing the efficiency of high-quality instruction data generation. Furthermore, the filtered first instruction data set can be used as part of the pre-training data, meaning a large amount of high-quality instruction data can be added to the pre-training data. This multi-task instruction data allows the model to learn richer language knowledge and task skills during the pre-training phase, thereby improving its generalization ability when handling unseen tasks or data samples and enhancing the model's performance in zero-shot and few-shot scenarios. Additionally, the filtered first instruction data set can be used as fine-tuning training data for text conversion tasks, helping to improve the model's bidirectional text conversion performance.

[0092] In some embodiments, the method for obtaining the first training data and the second training data includes:

[0093] Step S31: Obtain a set of text conversion data pairs, wherein the set of text conversion data pairs includes multiple spoken texts and multiple written texts corresponding to the spoken texts.

[0094] Electronic devices can directly obtain publicly available text conversion data pairs via the network, or use self-annotated text conversion data pairs, or obtain text conversion data pairs from a first set of instruction data pairs, or obtain text conversion data pairs from a second set of instruction data pairs.

[0095] Specifically, the text conversion data set in Table 1 below includes two samples, each of which includes a spoken text and the corresponding written text.

[0096] Table 1. Example of a text conversion data pair set provided in this application embodiment.

[0097]

[0098] Step S32: Construct the first training data based on the text conversion data set and the instructions for converting spoken text into written text.

[0099] Step S33: Construct the second training data based on the text conversion data set and the instructions for converting written text into spoken text.

[0100] In steps S32 and S33, different conversion instructions are added to the text conversion data set to construct the first training data and the second training data. That is, the text conversion data set is processed into the input text format for training the large model, as shown in Table 2 below. In Table 2, A and B represent the spoken and written text of each sample in step 1, respectively.

[0101] In this embodiment, different training data are constructed by adding different conversion instructions to the text conversion data pair set. This allows the first training data and the second training data to share the same text conversion data pair set. Furthermore, by adding different conversion instructions to construct the training data, it is convenient to train the large model to learn the conversion rules between spoken and written text, thereby improving its ability to convert spoken text into written text and vice versa.

[0102] Table 2. An example of input data provided in the embodiments of this application.

[0103]

[0104]

[0105]

[0106] In some embodiments, the first major model includes an expanded vocabulary for segmenting the input text, and the method further includes:

[0107] Step S101: Obtain the word segmentation training corpus.

[0108] The word segmentation training corpus can use high-quality open-source datasets such as Wudao, Chinese Wikipedia, Shusheng Wanjuan Text Dataset, and ChineseWebText. Specifically, a 3TB word segmentation training corpus can be used, with the data format being one sentence per line.

[0109] Step S102: Train the word segmentation model using the word segmentation training corpus to obtain an expanded vocabulary.

[0110] The word segmentation model can be the SentencePiece tool. SentencePiece is an open-source natural language processing tool used for word segmentation tasks in text preprocessing. It can use the BPE algorithm to segment text in various languages. The BPE algorithm is a data compression algorithm that compresses text by encoding the most frequently occurring substrings. In word segmentation tasks, the BPE algorithm can break down long words into multiple subwords, reducing the vocabulary size, increasing the model's generalization ability, and helping to improve model performance.

[0111] Specifically, set the size of the extended vocabulary to 20,000, specify the list of languages supported by the model as Chinese and English. To be consistent with the tokenization method of the first large model (such as LLaMA 2), the parameter split_digits in SentencePiece can be set to True, and the parameter byte_fallback can be set to True. Among them, the parameter split_digits is used to specify whether to split all numeric characters into separate units. For example, "2023" is split into independent sub-word units such as "2", "0", "2", "3". The advantage is that it can represent all numbers and reduce the number of numbers in the vocabulary. The disadvantage is that the number of individual units (tokens) will increase. For example, a "2023" is 4 tokens. The parameter byte_fallback is relatively important. It is used to specify whether to decompose unknown or rarely occurring characters into UTF-8 bytes for representation. If the parameter byte_fallback is not enabled, words that appear too few times in the training corpus will not be corresponding in the vocabulary. For example, assume that "chi mei wang liang" appears too few times in the training corpus. Then, without enabling the parameter byte_fallback, subsequent tokenization of "chi mei wang liang" cannot be performed. In this embodiment, the parameter byte_fallback is enabled, and subsequent tokenization of words that appear too few times in the training corpus can be performed. For example, "chi mei wang liang" will be tokenized using the UTF-8 encoding and 12 tokens such as "0xE9 0xAD 0x91 0xE90xAD 0x85 0xE9 0xAD 0x8D 0xE9 0xAD 0x89" can be obtained.

[0112] In this step, the above parameters are used for training the tokenization model. Sentencepiece will calculate the most frequently occurring substrings in the text and encode them. This encoding process is an iterative process until the training conditions are met (such as the vocabulary size reaches the set value), and an extended vocabulary with a size of 20,000 can be obtained.

[0113] Step S103: Obtain the initial vocabulary of the first large model, and merge the extended vocabulary and the initial vocabulary to obtain a merged vocabulary.

[0114] The initial vocabulary is the native vocabulary that the first large model has during the training process, and it is used for the first large model to perform tokenization on the input text. For example, when the first large model uses the LLaMA 2 model, the LLaMA 2 model has an initial vocabulary with a size of 32,000. This initial vocabulary is obtained by training the LLaMA 2 model with a large amount of data and contains rich semantic information.

[0115] A merged vocabulary is obtained by merging the expanded vocabulary and the initial vocabulary. This merged vocabulary retains the original vocabulary of the first major model and adds words that are not in the original vocabulary.

[0116] The original vocabulary of the first model contains only a small number of Chinese characters. When segmenting Chinese characters using the original vocabulary, a single Chinese character is often split into multiple tokens, requiring 2-3 tokens to form a single character, significantly reducing encoding and decoding efficiency. Furthermore, insufficient learning of languages ​​that did not appear or appeared very rarely during pre-training also affects the accuracy of model generation. In this embodiment, an expanded vocabulary is generated using the SentencePiece tool, and this expanded vocabulary is merged with the original vocabulary of the first model to obtain a merged vocabulary. The subsequent large model uses this merged vocabulary to segment the input text, expanding the vocabulary of Chinese characters, improving encoding and decoding efficiency, and enabling segmentation of words that appear too infrequently in the training corpus, thus improving the accuracy of model generation.

[0117] In some embodiments, the first large model includes an encoding module, and the training method includes: step S104: when pre-training the first large model, adding bias training parameters to the query matrix, key matrix and value matrix in the encoding module for pre-training.

[0118] Specifically, the encoding module may include multiple transformer decoder layers connected in sequence. Each transformer decoder layer has a trainable query matrix, key matrix, and value matrix. Trainable bias training parameters can be added to the query matrix, key matrix, and value matrix of each transformer decoder layer for pre-training. Then, during the pre-training process of the first large model, the large model will also learn the values ​​of each bias training parameter.

[0119] In this embodiment, pre-training by adding a bias term can achieve the effect of localized attention. Since attention localization is the key to the extrapolation of language models, it can improve the extrapolation ability of text conversion models.

[0120] In some embodiments, the second major model includes a first mapping layer for mapping the segmented input text, and the method further includes:

[0121] Step S21: When fine-tuning the second large model, add noise to the output of the first mapping layer and then fine-tune the training.

[0122] The structure of the second major model should be largely similar to that of the first major model. That is, the second major model also includes a mapping module, an encoding module, and a classification module connected in sequence. The mapping module includes a first mapping layer for mapping the segmented input text and a second mapping layer for mapping the position of the segmented input text. Specifically, the first mapping layer includes a token embedding layer, which converts the segmented input text into fixed-dimensional word vectors. The segmented input text can be a sequence of basic units (token sequences) that can be processed by the second major model. The second mapping layer includes a position embedding layer, which converts the position information of the segmented input text into positional codes. The second major model can then capture the order information of the segmented input text within the sequence based on these positional codes. The encoding module can include multiple transformer decoder layers connected in sequence. This encoding module converts the output of the mapping module into the corresponding token codes. The classification module can use the softmax function to convert the output of the encoding module into a probability distribution. It can be understood that the output of this encoding module should be the output of the last transformer decoder layer. The text generation process of the second major model is similar to that of the first major model and will not be elaborated upon here.

[0123] In this embodiment, the structural difference between the second and first large models lies in the fact that the second large model also includes a noise superposition layer. This noise superposition layer is used to superimpose noise onto the output vector of the token embedding layer and output the noise-superimposed output vector to the encoding module. That is, the output of the first mapping layer in the second large model is X′emb:

[0124]

[0125] Here, Xemb is the output of the token embedding layer in the second largest model, and Uniform() represents the uniform distribution function. is an adjustable parameter, L is the input length, and d is the dimension of the token embedding layer in the second largest model.

[0126] In this embodiment, the difference from the text generation process of the first large model lies in the noise superposition layer in the second large model. This layer superimposes uniformly distributed noise onto the output of the token embedding layer in the first mapping layer of the second large model before outputting it to the encoding module. The sampling range of this noise is... That is, a noise superposition layer is added to the second large model to fine-tune the structure of the second large model. The output of the token embedding layer in the first mapping layer of the second large model is superimposed with noise and then output to the encoding module. This improves the effect of instruction fine-tuning and can improve the adaptability of the subsequent text conversion model to small changes in the input, thereby improving the robustness and generalization ability of the text conversion model.

[0127] In some embodiments, the second major model also includes a second mapping layer for mapping the positions of the segmented input text, and the method further includes:

[0128] Step S22: When fine-tuning the second model, if the position of the input text after word segmentation exceeds the total number of position codes in the second mapping layer, then linear interpolation is used to increase the total number of position codes.

[0129] Typically, the second-largest model has a predefined context window size that matches the total number of positional encodings in the positional embedding layer of the second mapping layer, such as 2048. This means that the positional embedding layer in the second mapping layer can only encode the positions of 2048 tokens. When performing long-context tasks, if the input text exceeds 2048 tokens, the second-largest model will not be able to be trained.

[0130] In this embodiment, in order to accommodate more input tokens, positional encoding is interpolated between adjacent integer positions to obtain positional encodings corresponding to non-integer positions. This increases the total number of positional encodings in the positional embedding layer of the second large model, enabling the model to adapt to a longer context window. Compared to training a large model with a long context window from scratch, this embodiment increases the total number of positional encodings by linearly interpolating the positional encodings, thus enabling the model to adapt to a longer context window and reducing the time and cost spent.

[0131] The text conversion method provided in this application is described below with reference to exemplary applications and embodiments of the terminal provided in the embodiments of this application. See also: Figure 5 , Figure 5 This is a flowchart illustrating the text conversion method provided in this application embodiment. The method can be applied to electronic devices and includes the following steps:

[0132] Step S100: Obtain the first text to be converted and the conversion instructions.

[0133] The electronic device can extract the first text to be converted from audio and video information. This audio and video information can be obtained from a device equipped with a microphone or from a server. Alternatively, the electronic device may also have an interface for the user to input the first text to be converted, allowing the user to input the first text to be converted into the interface of the electronic device, which will then be processed by the processor of the electronic device.

[0134] The electronic device also features an interface for users to input conversion commands. Users can input these commands into the interface, allowing the device's processor to receive them. The conversion commands instruct the text conversion model on the direction of the conversion. For example, if the command is for converting spoken text to written text, it instructs the model to convert the first text to be converted into written text; if the command is for converting written text to spoken text, it instructs the model to convert the first text to be converted into spoken text.

[0135] Step S200: Input the first text to be converted and the conversion instruction into the text conversion model to obtain the first converted text, wherein the text conversion model is trained using the training method of any embodiment of the first aspect.

[0136] The electronic device has a built-in text conversion application, and the text conversion model is encapsulated in the text conversion application. The text conversion model is called to perform text conversion on the first text to be converted based on the aforementioned conversion instructions. After a series of calculations and processing, the first converted text is output.

[0137] It is understood that this text conversion model is trained using the method described in the above embodiments, and has the same structure and function as the text conversion model described in the above embodiments, so it will not be described in detail here.

[0138] In this embodiment, the text conversion model trained using the training method described in any of the above embodiments is used to convert the first text to be converted, thereby achieving the purpose of converting spoken text into written text and vice versa.

[0139] In some embodiments, the first text to be converted and the conversion instructions are input into the text conversion model to obtain the first converted text, including:

[0140] Step S210: Based on the first text to be converted, determine whether there is different identity information.

[0141] Identity information can be used to determine the speaker's role in the first text to be converted. For example, in a conference scenario, the speaker's identity can be determined based on the identity information; in a live streaming scenario, the host or assistant host role can be determined based on the identity information.

[0142] Specifically, the electronic device can determine whether different identity information exists from the sending end of the text. For example, if the first text to be converted contains text sent by two different sending ends, then it can be determined that there are two different identity information in the first text to be converted, and the text corresponding to the different identity information can be determined based on the different sending ends. Alternatively, if the first text to be converted contains text with identity information, then the electronic device can directly determine whether different identity information exists based on the first text to be converted.

[0143] Step S220: If different identity information exists, the first text to be converted is segmented according to the different identity information to obtain multiple second texts to be converted; if different identity information does not exist, the first text to be converted is segmented to obtain multiple second texts to be converted; or, the first text to be converted is not processed and is used as the second text to be converted.

[0144] Specifically, if the first text to be converted contains at least two identity information, the first text to be converted is divided into second texts corresponding to different identity information, and these second texts to be converted are arranged in chronological order.

[0145] If the first text to be converted does not contain identity information, it can be left unprocessed and directly used as the second text to be converted. Alternatively, the first text to be converted can be segmented into multiple second texts according to preset rules, and these second texts can be arranged in chronological order. The preset rules can be based on the maximum allowed time for each text; if the first text exceeds the maximum allowed time, it is segmented according to the maximum allowed time, ensuring that the time corresponding to each second text is less than the maximum allowed time. Alternatively, the preset rules can be based on the maximum allowed number of characters for each text; if the first text exceeds the maximum allowed number of characters, it is segmented according to the maximum allowed number of characters, ensuring that the number of characters corresponding to each second text is less than the maximum allowed number of characters.

[0146] Step S230: Input the second text to be converted and the conversion instruction into the text conversion model to obtain the first converted text.

[0147] Specifically, the electronic device calls the text conversion model to perform text conversion on each of the second texts to be converted based on the aforementioned conversion instructions. After a series of calculations and processing, the first converted text is output.

[0148] In this embodiment, the first text to be converted is segmented based on identity information or segmented based on preset rules, which can reduce the processing pressure on the text conversion model.

[0149] In some embodiments, step S230 includes:

[0150] Step S231: If the conversion instruction is to convert written text into spoken text, then input the second text to be converted and the conversion instruction into the text conversion model to obtain the second converted text.

[0151] After completing steps S210-S220, the electronic device will determine the conversion direction of the conversion instruction. If the conversion instruction is to convert written text into spoken text, the second text to be converted and the conversion instruction will be directly input into the text conversion model. The text conversion model will convert the second text to be converted into the corresponding spoken text, which is the second converted text.

[0152] Step S232: If the conversion instruction is to convert spoken text into written text, determine whether the second text to be converted needs to be filtered. If so, use the empty text as the second text to be converted. If not, input the second text to be converted and the conversion instruction into the text conversion model to obtain the second text to be converted.

[0153] If the conversion instruction is to convert spoken text into written text, the electronic device will use filtering rules or a binary classification model to determine whether to filter the second text to be converted. Here, empty text refers to text without content.

[0154] The filtering rules can be based on a user-defined set of meaningless text. If the second text to be converted contains words from this set of meaningless text, then the second text to be converted needs to be filtered, and the empty text will be used as the second converted text. If the second text to be converted does not contain words from this set of meaningless text, then no filtering is required. Specifically, this set of meaningless text can include texts such as "good" and "yes". If the second text to be converted contains "good" or "yes", then the empty text will be used as the second converted text.

[0155] A binary classification model is a machine learning model used to divide input data into two different categories. Specifically, a binary classification model can be trained with a large number of known meaningful and meaningless text samples, enabling it to learn the differences between meaningful and meaningless text. This allows the trained binary classification model to determine whether the second text to be converted belongs to the category of meaningless text that needs filtering or the category of meaningful text that does not need filtering. Alternatively, a binary classification model with the above functionality can be invoked to determine the category of the second text to be converted.

[0156] Step S233: Determine whether the second converted text needs to be merged. If so, obtain the second converted text to be merged and merge the second converted text to be merged to obtain the first converted text. If not, use the second converted text as the first converted text.

[0157] After obtaining the second converted text, it is necessary to determine whether the corresponding second text to be converted has been segmented or partitioned. If so, the second converted text to be merged needs to be obtained, and the two second converted texts are merged in chronological order to form the first converted text. If not, the second converted text is directly used as the first converted text.

[0158] In this embodiment, when converting spoken text into written text, the text to be converted is pre-determined to determine whether filtering is required. If so, it is filtered, and the first converted text is obtained after merging. This reduces the processing pressure on the text conversion model.

[0159] In some embodiments, the method further includes:

[0160] Step S300: Simultaneously display the first text to be converted and the first converted text on the display interface; and / or, display the first text to be converted on the display interface, and display added / deleted text annotations or identity information on the first text to be converted based on the first converted text.

[0161] Taking spoken text as the first text to be converted and written text as the first text to be converted, see [reference needed]. Figure 6 The system can simultaneously display the text before and after conversion on the display interface S1, allowing users to clearly compare the differences between the two. This helps users intuitively understand the changes in the text and better grasp the effect of the text conversion, so as to verify the accuracy and smoothness of the text conversion.

[0162] Adding and deleting text annotations includes one of the following: adding text annotations, deleting text annotations, and selecting text annotations. Adding text annotations marks text added to the first converted text relative to the first text to be converted; deleting text annotations marks text deleted from the first converted text relative to the first text to be converted; and selecting text annotations indicates whether the first converted text is selected. For example... Figure 7 or Figure 8 As shown, the first text to be converted can be displayed on the display interface S1. Added text annotations mark text added to the first text to be converted relative to the first text to be converted, and deleted text annotations mark text removed from the first text to be converted relative to the first text to be converted. In this case, Figure 7 In the text, adding and deleting text annotations uses different colors to highlight the text. Figure 8In the text annotation, adding a text annotation will change it to an underline, and deleting a text annotation will change it to a strikethrough. Selecting a text annotation can be... Figure 8 The speaker-level selection boxes and the all-selection box shown indicate that if a user selects the selection box corresponding to speaker 1, the text of speaker 1 is selected; if the user selects the all-selection box, all text is selected. The converted text is visualized based on added and deleted text annotations, and the differences between the first converted text and the first text to be converted are directly displayed on top of the first text to be converted. This helps users understand the changes in the text. In this way, users can clearly see which parts have been modified, added, or deleted, which is helpful for comparing version differences or for subsequent text editing.

[0163] Furthermore, such as Figure 8 As shown, the first-transformed text can be displayed according to the identity information of different speakers. By associating text with specific speakers through identity information, the dialogue or statements between different speakers are presented. Displaying text based on speaker identity information makes it easier for readers to understand who is saying what, thus improving their comprehension of the text content. This can be achieved through text color schemes and the use of different identifiers (such as...). Figure 8 The use of speakers (such as Speaker 1, Speaker 2, etc.) or identifiers to represent different identity information helps improve the clarity and coherence of text reading and facilitates its application in scenarios such as meeting minutes, interview transcription, and text display of multi-person dialogues.

[0164] In this embodiment, visualizing the text before and after conversion helps users better understand and process the results of text conversion, thereby improving the efficiency and accuracy of text reading.

[0165] This application also provides a computer-readable storage medium storing computer-executable instructions for causing an electronic device to execute the method for training a text conversion model or the text conversion method provided in this application.

[0166] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0167] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0168] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0169] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0170] This application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the method for training a text conversion model or the text conversion method as described in the foregoing embodiments.

[0171] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for at least one computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a text conversion model, characterized in that, include: Acquire pre-training data, and pre-train the first large model based on the pre-training data to obtain the second large model; Obtain fine-tuning training data, and fine-tune the second major model based on the fine-tuning training data to obtain the text conversion model. The fine-tuning training data includes first training data for converting spoken text into written text and second training data for converting written text into spoken text.

2. The training method according to claim 1, characterized in that, The acquisition of pre-training data includes: Get the text of the instruction data to be generated; Construct an input template for generating instruction-type data; Based on the text of the instruction data to be generated and the input template, a first set of instruction data pairs is obtained; The first instruction data pair set is evaluated and filtered to obtain the filtered first instruction data pair set. Retrieve the second set of instruction data pairs and unlabeled data; The pre-training data is obtained based on the filtered first set of instruction data pairs, the second set of instruction data pairs, and the unlabeled data.

3. The training method according to claim 1, characterized in that, The method for obtaining the first training data and the second training data includes: Obtain a set of text-to-text conversion data pairs, wherein the set of text-to-text conversion data pairs includes multiple spoken texts and written texts corresponding to the multiple spoken texts; Based on the text conversion data set and the instructions for converting spoken text into written text, the first training data is constructed; The second training data is constructed based on the text conversion data set and the instructions for converting written text into spoken text.

4. The training method according to claim 1, characterized in that, The first major model includes an expanded vocabulary for segmenting the input text, and the training method further includes: Obtain training corpus for word segmentation; The word segmentation model is trained using the aforementioned word segmentation training corpus to obtain an expanded vocabulary; Obtain the initial vocabulary of the first large model, and merge the expanded vocabulary and the initial vocabulary to obtain the merged vocabulary.

5. The training method according to claim 1, characterized in that, The first large model includes an encoding module, and the training method includes: During the pre-training of the first large model, bias training parameters are added to the query matrix, key matrix, and value matrix in the encoding module for pre-training.

6. The training method according to any one of claims 1-5, characterized in that, The second major model includes a first mapping layer for mapping the segmented input text, and the training method further includes: When fine-tuning the second large model, noise is superimposed on the output of the first mapping layer before fine-tuning.

7. The training method according to any one of claims 1-5, characterized in that, The second major model also includes a second mapping layer for mapping the positions of the segmented input text, and the training method further includes: When fine-tuning the second model, if the position of the input text after word segmentation exceeds the total number of position codes in the second mapping layer, then linear interpolation is used to increase the total number of position codes.

8. A text conversion method, characterized in that, include: Obtain the first text to be converted and the conversion instructions; The first text to be converted and the conversion instruction are input into the text conversion model to obtain the first converted text, wherein the text conversion model is trained using the training method described in any one of claims 1-7.

9. The method according to claim 8, characterized in that, The step of inputting the first text to be converted and the conversion instruction into the text conversion model to obtain the first converted text includes: Based on the first text to be converted, determine whether there is different identity information; If different identity information exists, the first text to be converted is segmented according to the different identity information to obtain multiple second texts to be converted; If there is no different identity information, the first text to be converted is segmented to obtain multiple second texts to be converted; Alternatively, the first text to be converted may not be processed, and the first text to be converted may be used as the second text to be converted. The second text to be converted and the conversion instruction are input into the text conversion model to obtain the first converted text.

10. The method according to claim 9, characterized in that, The step of inputting the second text to be converted and the conversion instruction into the text conversion model to obtain the first converted text includes: If the conversion instruction is an instruction to convert written text into spoken text, then the second text to be converted and the conversion instruction are input into the text conversion model to obtain the second converted text; If the conversion instruction is an instruction to convert spoken text into written text, then it is determined whether the second text to be converted needs to be filtered. If so, then the empty text is used as the second converted text. If not, then the second text to be converted and the conversion instruction are input into the text conversion model to obtain the second converted text. Determine whether the second converted text needs to be merged. If so, obtain the second converted text to be merged and merge the second converted text to be merged to obtain the first converted text. If not, use the second converted text as the first converted text.

11. The method according to any one of claims 8-10, characterized in that, The method further includes: The first text to be converted and the first text to be converted are displayed simultaneously on the display interface; And / or, display the first text to be converted on the display interface, and display added or deleted text annotations or identity information on the first text to be converted based on the first text to be converted.

12. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 11.

Images