A sentence generation method and related device

Abstract

The application relates to the field of artificial intelligence and discloses a word and sentence generation method, which comprises the following steps: obtaining a target character string sequence, the target character string being input by a user in an input method tool; generating a target word and sentence corresponding to the target character string sequence through a target neural network according to the target character string sequence, wherein the target neural network comprises an encoder and a decoder, the encoder is used for obtaining an embedding vector according to the target character string sequence, the decoder is used for generating the target word and sentence according to the embedding vector, the target neural network is obtained by training a training sample, and the training sample comprises a character string sequence and a corresponding word and sentence; and presenting the target word and sentence in an interface of the input method tool. The application inputs a character string without error correction and word segmentation into the target neural network, reduces the error superposition influence of a word and sentence precision caused by an error correction model and a word segmentation model, and improves the generation precision of the word and sentence.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a word and sentence generation method and related equipment. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Input method editors are essential client applications, widely used in desktops, laptops, mobile phones, tablets, smart TVs, and in-vehicle computers. Furthermore, users' daily activities, such as searching for locations, finding restaurants, chatting with friends, and planning trips, largely translate into input behavior. Therefore, data from input method editors can be used to accurately profile users. Consequently, input method editors hold significant strategic importance in the internet field.

[0004] In input scenarios, after a user enters characters (such as Pinyin) on a device, the input method editor generates words or phrases and prompts the user to select from them. The accuracy of the generated words or phrases directly affects the accuracy of the input method editor and the user experience; therefore, a method that can accurately generate words or phrases is needed. Summary of the Invention

[0005] This application provides a word and sentence generation method that reduces the cumulative impact of errors from error correction models and word segmentation models on word and sentence accuracy, thereby improving the accuracy of word and sentence generation.

[0006] Firstly, this application provides a word / sentence generation method, the method comprising:

[0007] Obtain the target string sequence, wherein the target string is entered by the user in the input method tool;

[0008] In one possible implementation, a string can be understood as a combination of characters, a carrier of linguistic information used to generate words and phrases; these words and phrases can be a single word, multiple words, or even a single character. Users can input a target string sequence using an input method tool, and then the terminal device can obtain the user-inputted target string sequence.

[0009] Based on the target string sequence, a target word / phrase corresponding to the target string sequence is generated through a target neural network. The target neural network includes an encoder and a decoder. The encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target word / phrase based on the embedding vector. The target neural network is trained using training samples, which include string sequences and corresponding word / phrase (e.g., the correct word / phrase corresponding to the string sequence). The target word / phrase is then presented in the interface of the input method tool.

[0010] In one possible implementation, the encoder can obtain an embedding vector from the target string sequence, where the encoder can process each character in the target string sequence to obtain an embedding vector (or latent vector) for each character. It should be understood that the size of the encoder's input and output can be kept consistent.

[0011] In one possible implementation, the decoder can generate the target phrase based on the embedding vector. Optionally, the decoder can obtain at least one word unit and the probability of each word unit based on the embedding vector, and combine this with a planning algorithm to obtain the target phrase.

[0012] In existing implementations, strings input from input method tools require preprocessing through error correction and word segmentation modules before being input into the phoneme-to-character conversion model. In this embodiment, the uncorrected and unsegmented string is directly input into the phoneme-to-character conversion model (e.g., the target neural network in this embodiment). During training of the target neural network, noisy strings can be used directly, enabling the target neural network to have error correction capabilities. Furthermore, since the target string sequence is obtained through input from an input method tool, the character length is not very long (less than a threshold). Even without word segmentation, the target neural network can still obtain accurate words and sentences directly from the original string. This solves the problem of the cumulative error impact of the error correction model and word segmentation model on the accuracy of words and sentences in existing technologies, thus improving the accuracy of word and sentence generation.

[0013] Furthermore, by training the target neural network with the aforementioned noisy samples, the target neural network can acquire error correction capabilities (that is, the target neural network can still generate correct words and sentences even for strings containing noise).

[0014] In one possible implementation, the number of characters in the target string sequence is less than a threshold, which is a value less than or equal to 128. For example, the threshold can be 64, 70, 80, 90, 100, 128, etc.

[0015] In one possible implementation, the decoder can employ a non-autoregressive parallel decoding approach. First, in the key-phoneme conversion problem, the input is a sequence of letters, and the output is a sequence of Chinese characters. Typically, one Chinese character needs to be represented by multiple letters, so the length of the output Chinese character sequence is usually much shorter than the input letter sequence. Therefore, a 'generated sequence length prediction' module is added to the encoder to guide the length of the generated sequence. Second, the decoder is changed from unidirectional Attention (e.g., the GPT model) to bidirectional Attention using the BERT model to support parallel decoding.

[0016] In one possible implementation, the target phrase may include a first character unit and a second character unit, wherein the first character unit is positioned earlier in the target phrase than the second character unit, and the decoder is specifically used to generate the second character unit based on the target string sequence without relying on the fact that the first character unit has already been generated.

[0017] In one possible implementation, the decoder is specifically used to: generate the first word unit and the second word unit in parallel based on the target string sequence.

[0018] In this embodiment of the application, non-autoregressive decoding can significantly improve the inference speed of the model while maintaining the model's performance without a significant decrease.

[0019] In one possible implementation, the number of character units in the target word / phrase can be predicted using a character count prediction model based on the target string sequence; an initial word / phrase corresponding to the target string sequence can be generated using a target neural network based on the target string sequence; and the initial word / phrase can be truncated based on the number of character units to obtain the target word / phrase.

[0020] Specifically, after receiving the target string sequence, the target neural network can encode the input sequence through an encoder; predict the length of the target words (number of sub-units) through a word count prediction model; the decoder can generate the initial words corresponding to the target string sequence in parallel based on the encoder's encoding results; finally, the initial words are adjusted based on the previously predicted number of sub-units (e.g., truncating parts that exceed the length).

[0021] In one possible implementation, the word count prediction model can be either a classification model or a regression model.

[0022] In one possible implementation, to improve the accuracy of the target phrase, a word count prediction model can be used to predict the number of sub-units of the target phrase, and the initial phrase can be adjusted based on the number of sub-units.

[0023] In one possible implementation, the target string sequence is a string sequence containing noise, the noise being caused by user mis-input in the input method tool;

[0024] The target phrase is the correct phrase corresponding to the target string sequence after noise removal.

[0025] In one possible implementation, the target neural network includes an encoder and a decoder, wherein the encoder or decoder can be one of the following models: LSTM, GRU, SRU, BERT, Roberta, SpanBERT, XLNet, GPT, Nezha, Mass, Bart, MBART, Albert, StructBERT, Ernie, KnowBERT, K-BERT, TinyBERT.

[0026] In one possible implementation, the encoder can be understood as a deep learning network model. There are various network structures for the encoder, and this application does not specifically limit them. Specifically, the network structure of the encoder can adopt the network structure of the encoder part of the Transformer network, or adopt the network structure of a series of other networks obtained from the encoder part of the Transformer network.

[0027] Secondly, this application provides a sample construction method, the method comprising:

[0028] Obtain the first string sequence and its corresponding words, wherein the first string sequence includes the first character;

[0029] When constructing samples, since string sequences are difficult to obtain, in order to obtain the first string sequence, the word and sentence can be converted into the first string sequence through the phonetic conversion module.

[0030] In one possible implementation, the first character is a character in the first string sequence.

[0031] In one possible implementation, the first character can be obtained by randomly sampling (or otherwise) the characters of the first string sequence. The first character can also be an object that adds noise to the first string sequence (specifically, by replacing the first character with a character other than the first character, or by adding a character other than the first character before or after the first character).

[0032] A target probability model is used to determine the target character corresponding to the first character from at least one second character. The target probability model represents the probability that a user accidentally presses a virtual key corresponding to each of the at least one second character when inputting the first character on a virtual keyboard. This probability is related to at least one of the following:

[0033] The size information of the virtual buttons, the layout information of the virtual buttons, the user's operating habits, or the user's hand structure characteristics;

[0034] The first character in the first string sequence is replaced with the target character, or the first character is added before or after the target character to obtain a second string sequence. The second string sequence and the phrase are used as training samples for the target neural network, which is used to generate corresponding phrases based on the string sequence.

[0035] The target probability model can be used to characterize the probability that a user accidentally touches the virtual key corresponding to each of the at least one second characters when inputting the first character on the virtual keyboard.

[0036] In one possible implementation, the probability of a user accidentally pressing different buttons may not be equal. This probability can be related to factors such as the size and layout of the virtual buttons, the user's operating habits, or the user's hand structure characteristics.

[0037] In one possible implementation, the larger the size of the virtual button, the greater the probability of it being accidentally pressed. For example, when a user presses button A, and there are buttons B, C, and D nearby, if the size of button B is larger than the sizes of buttons C and D, then the probability of the user accidentally pressing button B while pressing button A is higher.

[0038] In one possible implementation, keyboards with different virtual key size information can correspond to different target probability models.

[0039] In one possible implementation, the layout information of the virtual keys may include the arrangement of the keys on the keyboard, the distance between the keys, and the shape of the keys themselves. For example, when a user presses key A, if key A is near keys B, C, and D, and the distance between key B and key A is less than the distance between key C and key D and key A, then the probability of the user accidentally pressing key B when pressing key A is higher.

[0040] In one possible implementation, a keyboard with layout information for different virtual keys can correspond to different target probability models.

[0041] In one possible implementation, user operating habits can be understood as the user's habitual actions when pressing keys. Different users may have different habits. For example, if key A is near keys B, C, and D, user A is more likely to accidentally press key B when pressing key A due to their operating habits. Thus, the probability of user A accidentally pressing key B when pressing key A is higher. Another example is that operating habits may be related to keyboard input proficiency.

[0042] In one possible implementation, users with different operating habits can correspond to different target probability models.

[0043] In one possible implementation, a user's hand structure features can be understood as the structural features of the fingers when pressing keys, such as the size of the area between the fingers and the contact surface. Hand structure features can be age-related; even for users of the same age, different hand structure features may correspond to different genders and individual differences.

[0044] In one possible implementation, users with different hand structure features can correspond to different target probability models.

[0045] In one possible implementation, the size information of the virtual button may include the size information of at least one second character.

[0046] In one possible implementation, the layout information of the virtual key may include at least one layout feature between the second character and the first character.

[0047] In this embodiment, the target character used to replace the first character is determined by the target probability model, which can more accurately depict the user's actual behavior, that is, determine the character that is more likely to be accidentally touched. As a result, the noise-added training samples obtained can better reflect the actual user operation. Consequently, the target neural network trained based on the noise-added training samples is more accurate, thereby enhancing the robustness of the model in real user input scenarios.

[0048] In one possible implementation, the method further includes:

[0049] Obtain the dot cloud of key presses on the virtual keyboard when a sample user inputs characters;

[0050] Based on the point cloud of the pressed points, the target probability model is constructed.

[0051] Since the point cloud of press dots can characterize the user's operating habits, the probability determined by the target probability model built based on the point cloud of press dots can be related to the user's operating habits.

[0052] In one possible implementation, the target probability model is a Gaussian probability model.

[0053] In one possible implementation, the method further includes:

[0054] The target neural network is trained based on the second string sequence and the correct words / phrases.

[0055] Thirdly, this application provides a word and sentence generation apparatus, the apparatus comprising:

[0056] The acquisition module is used to acquire a target string sequence, wherein the target string is entered by the user in the input method tool;

[0057] A word / phrase generation module is used to generate target words / phrase corresponding to the target string sequence through a target neural network based on the target string sequence. The target neural network includes an encoder and a decoder. The encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target words / phrase based on the embedding vector. The target neural network is trained using training samples, which include string sequences and corresponding words / phrase.

[0058] The presentation module is used to present the target words and phrases in the interface of the input method tool.

[0059] In one possible implementation, the number of characters in the target string sequence is less than a threshold, which is a value less than or equal to 128.

[0060] In one possible implementation, the target phrase includes a first character unit and a second character unit, wherein the first character unit is positioned earlier in the target phrase than the second character unit, and the decoder is specifically used to generate the second character unit based on the target string sequence without relying on the fact that the first character unit has already been generated.

[0061] In one possible implementation, the decoder is specifically used to: generate the first word unit and the second word unit in parallel based on the target string sequence.

[0062] In one possible implementation, the device further includes:

[0063] The word count prediction module is used to predict the number of character units of the target word / sentence based on the target string sequence using a word count prediction model.

[0064] The word and sentence generation module is specifically used for:

[0065] Based on the target string sequence, an initial phrase corresponding to the target string sequence is generated using a target neural network;

[0066] Based on the number of character units, the initial word / phrase is truncated to obtain the target word / phrase.

[0067] In one possible implementation, the target string sequence is a string sequence containing noise, which is caused by the user's erroneous input in the input method tool;

[0068] The target phrase is the correct phrase corresponding to the target string sequence after noise removal.

[0069] In one possible implementation, the encoder or decoder is one of the following models:

[0070] LSTM, GRU, SRU, bert, roberta, spanbert, xlnet, GPT, nezha, mass, bart, mbart, albert, structbert, ernie, knowbert, k-bert, tinybert.

[0071] Fourthly, this application provides a sample construction apparatus, the apparatus comprising:

[0072] The acquisition module is used to acquire a first string sequence and its corresponding words and phrases, wherein the first string sequence includes a first character;

[0073] A character replacement module is used to determine the target character corresponding to the first character from at least one second character using a target probability model, wherein the target probability model represents the probability that a user accidentally presses a virtual key corresponding to each of the at least one second character when inputting the first character on a virtual keyboard, and the probability is related to at least one of the following:

[0074] The size information of the virtual buttons, the layout information of the virtual buttons, the user's operating habits, or the user's hand structure characteristics;

[0075] The first character in the first string sequence is replaced with the target character, or the first character is added before or after the target character to obtain a second string sequence. The second string sequence and the phrase are used as training samples for the target neural network, which is used to generate corresponding phrases based on the string sequence.

[0076] In one possible implementation, the target probability model is a Gaussian probability model.

[0077] In one possible implementation, the device further includes:

[0078] The training module is used to train the target neural network based on the second string sequence and the correct words / phrases.

[0079] Fifthly, embodiments of this application provide a neural network search device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect and any optional methods described in the first aspect and any optional methods described in the second aspect.

[0080] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the first aspect and any optional method thereof, as well as the second aspect and any optional method thereof.

[0081] In a seventh aspect, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any optional method thereof, as well as the second aspect and any optional method thereof.

[0082] Eighthly, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. The chip system may be composed of chips or may include chips and other discrete devices.

[0083] This application provides a word / phrase generation method, comprising: acquiring a target string sequence, wherein the target string is input by a user in an input method tool; generating target words / phrases corresponding to the target string sequence through a target neural network, wherein the target neural network includes an encoder and a decoder, the encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target words / phrases based on the embedding vector, the target neural network is trained using training samples, wherein the training samples include string sequences and corresponding words / phrases; and presenting the target words / phrases in the interface of the input method tool. By inputting uncorrected and unsegmented strings into a phonetic-to-character conversion model (such as the target neural network in this embodiment), the method solves the problem of the cumulative error impact on word / phrase accuracy caused by the error correction model and the segmentation model in the prior art, thereby improving the accuracy of word / phrase generation.

[0084] It should be understood that the methods and apparatus described above can be referenced, combined, and interpreted in a way that does not contain any technical contradictions. Attached Figure Description

[0085] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0086] Figure 2 This is a schematic diagram of an input method interface.

[0087] Figure 3 This is a schematic diagram of an input method interface.

[0088] Figure 4 This is a schematic diagram of an application system architecture;

[0089] Figure 5 This is a schematic diagram of an application system architecture;

[0090] Figure 6 This is a schematic diagram of a word and sentence generation method;

[0091] Figure 7 This is a schematic diagram of a word and sentence generation method;

[0092] Figure 8 A flowchart illustrating the word and sentence generation method provided in the embodiments of this application;

[0093] Figure 9 This is a schematic diagram illustrating the construction of the embedded vector in an embodiment of this application;

[0094] Figure 10 This is a schematic diagram of a non-autoregressive network;

[0095] Figure 11 This is a schematic diagram of word and phrase generation in an embodiment of this application;

[0096] Figure 12 A flowchart illustrating the sample construction method provided in this application embodiment;

[0097] Figure 13 A schematic diagram of the click distribution when a user clicks on a virtual keyboard;

[0098] Figure 14 This is a schematic diagram of a sample construction method;

[0099] Figure 15 This application provides an embodiment of a word and phrase generation device.

[0100] Figure 16 This application provides an embodiment of a sample construction apparatus.

[0101] Figure 17 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0102] Figure 18 This is a schematic diagram of a training device provided in an embodiment of this application;

[0103] Figure 19 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0104] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0105] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0106] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0107] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0108] (1) Infrastructure

[0109] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0110] (2) Data

[0111] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0112] (3) Data processing

[0113] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0114] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0115] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0116] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0117] (4) General ability

[0118] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0119] (5) Smart Products and Industry Applications

[0120] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0121] This application can be applied, but is not limited to, the field of natural language processing within the field of artificial intelligence. Specifically, it can be applied to the field of natural language processing, and several application scenarios that have been implemented in products will be introduced below.

[0122] To better understand the solutions of the embodiments of this application, the following will first combine... Figures 2 to 4 A brief introduction to the possible application scenarios of the embodiments of this application is provided.

[0123] This application can be applied to scenarios where information is input based on an input method.

[0124] In this input scenario, the user can input a string on the terminal device. Correspondingly, the input method editor (IME) deployed inside the terminal device will receive the string input by the user, generate corresponding words and phrases based on the string, and then prompt the user with the words and phrases.

[0125] The input method editor can be implemented by a neural network, such as the target neural network in the embodiments of this application.

[0126] In this application embodiment, the task of converting a string into a corresponding word or phrase is referred to as a phonetic-to-character conversion task.

[0127] In one possible implementation, a string (also known as a string sequence, such as the target string sequence in the embodiments of this application) can be understood as a combination of characters, a carrier of linguistic information, used to generate words and phrases; the words and phrases can be a single word or multiple words, and a single character can also be a word.

[0128] In one possible implementation, the string can be characters representing the pronunciation of the words or phrases the user wants to input.

[0129] The above input scenarios can be input scenarios in multiple languages ​​such as Chinese, Japanese, and Mandarin; the form of the string is different for different languages; taking Chinese as an example, the string can include one or more pinyin.

[0130] Specifically, such as Figure 2 and Figure 3 As shown, when the string "nuoyafangzhou" is entered, the input method editor suggests the following phrases: Noah's Ark, Noah's Ark, Noah's Ark Continent, and Noah's Ark.

[0131] Reference Figure 2 , Figure 2The diagram illustrates an interface for input on a mobile terminal using an input method. Users can perform input operations such as entering text into an input box or editing text on a text editing page. The mobile terminal can receive these input operations. Here, the input operation can be a sequence of characters entered into a spelling area, and the input method can generate corresponding candidate words based on this sequence. For example, the input operation can be either keyboard input or handwriting input. Furthermore, the input operation can be other types of input operations, which are not limited in this embodiment.

[0132] Reference Figure 3 , Figure 3 This shows a schematic of the interface when inputting data using an input method on a PC terminal, and Figure 2 The difference is, Figure 2 Users can input strings using the virtual keyboard on the mobile terminal's display interface. Figure 3 Users can input strings using the physical keyboard.

[0133] In this embodiment, the terminal device can be a desktop computer, a laptop computer, a tablet computer, a smartphone, or a smart TV. In addition, the terminal device can also be any other device that can deploy an input method editor, such as an in-vehicle computer.

[0134] The application architecture of this application embodiment will be described next.

[0135] Reference Figure 4 , Figure 4 A natural language processing system is illustrated, which includes a user device (which may also be referred to as a terminal device or smart device in this application embodiment). The user device includes terminal devices such as mobile phones and personal computers.

[0136] exist Figure 4 In the system shown, the user equipment can receive user instructions, such as receiving a string input by the user, and then performing processing on the string (e.g., performing a phonetic-to-character conversion task) to obtain the corresponding processing result for the string (e.g., the words or phrases corresponding to the string).

[0137] In this embodiment of the application, the user equipment may store a target neural network and perform inference tasks (such as the aforementioned phonetic-to-character conversion task) based on the target neural network each time the operating system (OS) or application (APP) calls the model.

[0138] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0139] (1) Neural Network

[0140] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0141] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0142] (2) Deep Neural Networks

[0143] A Deep Neural Network (DNN) can be understood as a neural network with many hidden layers. There's no specific metric for "many" layers; the terms "multi-layer neural network" and "deep neural network" are essentially the same thing. DNNs can be categorized into three layers based on their position: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs seem complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number is quite large. So, how are the specific parameters defined in a DNN? First, let's look at the definition of the coefficient W. Taking a three-layer DNN as an example, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as... Note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks.

[0144] (3) Natural Language Processing (NLP)

[0145] Natural language is human language, and Natural Language Processing (NLP) is the processing of human language. NLP is a systematic process of analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform numerous automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering systems, and topic segmentation, among others.

[0146] (4) Loss Function

[0147] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0148] (5) Backpropagation algorithm

[0149] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0150] (6) Input method preferred word: When a user enters a string, the input method editor will provide the user with a candidate list. This candidate list is used to suggest words and phrases to the user. The first word in the candidate list is called the input method's preferred word.

[0151] (7) Input method error correction module: When users are inputting, there may be 'accidental touch phenomenon', that is, the intention is to press a certain key, but in fact another key is pressed. The input method error correction module is a module that corrects the incorrect key information to the correct key information.

[0152] (8) Pinyin segmentation module: The original sequence of input by the input method user is a sequence of letters without separation. The pinyin segmentation module segments the input sequence to form a sequence composed of complete pinyin, which is then sent to the phonetic-character conversion module.

[0153] (9) Transformer network structure: a deep neural network structure that includes substructures such as input layer, self-attention layer, feed-forward layer, and normalization layer.

[0154] (10) Bert model: A model with a Transformer network structure. Based on the Transformer network structure, a learning paradigm of "pre-training + fine-tuning" was proposed, and two pre-training tasks, Masked Language Model and NextSentence Prediction, were designed.

[0155] (11) N-gram model: a model that is widely used in Chinese input method tasks.

[0156] (12) Bart: Using the Bert model as the encoder and the GPT model as the decoder, and designing a variety of pre-training tasks to train the model, Bart has achieved good results in both understanding and generation tasks in NLP.

[0157] (13) Zero probability problem: In the process of using the Ngram model, in some cases, the probability value will be calculated as zero. Zero probability will cause many problems in engineering implementation; for example, because of zero probability, it is impossible to compare the size of the probabilities, and only the result can be returned randomly.

[0158] (14) Smoothing algorithm: An algorithm designed to solve the zero probability problem of Ngram models. When it is determined that there is a zero probability risk, the smoothing algorithm usually uses stable but inaccurate low-order Ngram model probabilities to fit unstable but accurate high-order Ngram model probabilities in some way.

[0159] (15) Viterbi algorithm: It is a dynamic programming algorithm used to find the Viterbi path, or hidden state sequence, that is most likely to produce the sequence of observed events, especially in the context of Markov information sources and hidden Markov models. It is now often used in speech recognition, keyword recognition, computational linguistics and bioinformatics. The Viterbi algorithm can also be called the Finite State Transducers (FST) algorithm.

[0160] The N-gram model will be introduced in detail below.

[0161] For a language sequence (e.g., a phrase is a word sequence), the probability P(w1, w2, ... w) of the sequence is... nP(w1, w2, ..., w) can be decomposed into a product of conditional probabilities, as follows: n )=P(w1)*P(w2|w1)*P(w3|w1,w2)*……P(w n |w1,……w n-1 ), where w1, w2……w n Let represent the words in the sequence, and P represent the probability.

[0162] However, it is difficult to accurately obtain the probability P(w) using statistical methods. n |w1,……w n-1 The value of ) is given. Therefore, the N-gram model makes the Markov assumption, that is, the probability of the current word is only related to a finite number of N words. When N takes different values, a series of specific N-gram models are obtained. For example: when N=2, the probability of the current word is only related to the previous word, P(w n |w1,……w n-1 The value of ) degenerates into P(w) n |w n-1 The value of ), that is The N-gram model at this point is called the Bigram model; similarly, when N=3, the N-gram model is called the Trigram model; and when N=4, the N-gram model is called the Fourgram model.

[0163] During use, the N-gram model has a problem. In application scenarios, some word combinations may not appear in the training set. In this case, N-gram estimates a probability of 0 for these word combinations, which can cause a series of problems in engineering. To avoid this 0-probability situation, various smoothing algorithms have been developed.

[0164] The smoothing algorithm can be simply understood as follows: when the probability of the N-gram model is 0, the product of a certain weight and the probability of the (N-1)-gram model is taken as the probability of the (N)-gram model.

[0165] The N-gram model will be illustrated with a specific example below.

[0166] Specifically, suppose the word sequence is: Noah's technology is strong; the probability of the word sequence can be decomposed into the product of conditional probabilities, that is, P(Noah, of, technology, skill, strong) = P(Noah) * P(Noah|Noah) * P(of|Noah,Asia) * P(technology|Noah,Asia, of) * P(skill|Noah,Asia, of, technology) * P(strong|Noah,Asia, of, technology, skill);

[0167] After adopting the gram model with N=2, P(Nuo, Ya, De, Ji, Shu, Qiang)=P(Nuo|B)*P(Ya|Nuo)*P(De|Ya)*P(Ji|De)*P(Shu|Ji)*P(Qiang|Shu);

[0168] After adopting the gram model with N=3, P(Nuo, Ya, De, Ji, Shu, Qiang)=P(Nuo|A, B)*P(Ya|Nuo, B)*P(De|Nuo, Ya)*P(Ji|Nuo, Ya, De)*P(Shu|Nuo, Ya, De, Ji)*P(Qiang|Nuo, Ya, De, Ji, Shu).

[0169] It should be noted that when N=1, since there is no other character before "Nuo", one character (represented by A in the above example) will be automatically added as a reference during the calculation process using the Ngram model; similarly, when N=2, since there is no other character before "Nuo", two characters (represented by A and B in the above example) will be automatically added as a reference during the calculation process using the Ngram model.

[0170] The Viterbi algorithm will be described below.

[0171] Take the pinyin input method as an example, as shown in Figure 6 , the bottom row represents pinyin nodes, and the nodes in the upper four rows are Chinese characters corresponding to the pinyin nodes, and these Chinese characters constitute various possibilities of user input. The Ngram model can be used to calculate the probability of each Chinese character node. Since the probability of a Chinese character node is actually a conditional probability under the condition that the previous N Chinese character nodes occur, this probability can also be regarded as the path transition probability between Chinese character nodes.

[0172] For example, when N=2, the Ngram model can be used to calculate the probabilities P(Ya|Nuo), P(Ya|Nuo<懦>), P(Ya|Nuo<糯>), P(Ya|Nuo<挪>), and these probabilities can also be called the path transition probability from "Nuo<诺>" to "Ya", the path transition probability from "Nuo<懦>" to "Ya", the path transition probability from "Nuo<糯>" to "Ya", and the path transition probability from "Nuo<挪>" to "Ya".

[0173] Corresponding to each of the six pinyins "nuo", "ya", "de", "ji", "shu", "qiang", there are four choices for the Chinese character, so the number of these Chinese character combinations is 4*4*4*4*4*4; by using the Viterbi algorithm and the path transition probability between Chinese characters, a node path with the maximum probability can be searched, this node path can also be called the optimal path, which can specifically be Figure 6 "Nuoya de jishu qiang" shown in .

[0174] Next, a more detailed architecture of the execution subject for executing the phrase generation method in the embodiments of the present application is introduced.

[0175] The following is combined with Figure 5 The system architecture provided in the embodiments of this application will be described in detail. Figure 5 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 5 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0176] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0177] The data acquisition device 560 is used to collect training samples. In this embodiment, the training samples can be data used when training the neural network (e.g., strings and corresponding words). After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0178] The training device 520 can train the neural network based on training samples to search for the target model / rule 501. In this embodiment, the target model / rule 501 can be the target neural network.

[0179] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0180] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 5 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc.

[0181] Specifically, the training device 520 can transmit the target neural network to the execution device 510.

[0182] exist Figure 5In the process, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. Users can input data (such as a target string sequence in this embodiment) into the I / O interface 512 through the client device 540.

[0183] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0184] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0185] Finally, the I / O interface 512 presents the processing results (e.g., the target words / phrases in this embodiment) to the client device 540, thereby providing them to the user.

[0186] exist Figure 5 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various ways such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0187] It is worth noting that, Figure 5 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 5In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0188] From the inference side of the model:

[0189] In this embodiment, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the word and sentence generation method in this embodiment.

[0190] In this embodiment of the application, the computing module 511 of the execution device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0191] Specifically, the computing module 511 of the execution device 520 can be a hardware system with the function of executing instructions. The word and sentence generation method provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the word and sentence generation method provided in this application embodiment.

[0192] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the word and sentence generation method provided in the embodiments of this application can also be implemented by the hardware system without the function of executing instructions in the computing module 511 of the execution device 520, which is not limited here.

[0193] From the training side of the model:

[0194] In this embodiment of the application, the training device 520 can access the memory ( Figure 5 The methods related to model training in the embodiments of this application can be implemented by code stored in the training device 520 (which may be integrated into the training device 520 or deployed separately from the training device 520).

[0195] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0196] Specifically, the training device 520 can be a hardware system with instruction execution capabilities. The model training-related methods provided in this application embodiment can be software code stored in memory. The training device 520 can retrieve the software code from memory and execute the retrieved software code to implement the model training-related methods provided in this application embodiment.

[0197] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the model training-related methods provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0198] Existing input method tools can generate corresponding candidate words based on the user's string input and display them on the input method tool interface for the user to select. (See reference...) Figure 7 Taking a string as an example of Pinyin, the input method software needs to go through three steps from the user's string input to the input method generating candidate words:

[0199] 1. Error correction

[0200] Because users' original input often contains errors, input method software first corrects these errors. Specifically, users are prone to 'accidental key presses' when actually typing, meaning they intend to press one key but actually press another. The keyboard correction module converts the user's actual input sequence into a correct key sequence that matches the user's input intention. For example... Figure 7As shown, the user's actual input sequence was 'nuiyafangzou', where 'nuo' was incorrectly entered as 'nui' (because 'o' and 'i' are very close on the keyboard), and the retroflex consonant 'zhou' was incorrectly entered as 'zou'. After the keyboard correction module, the sequence was corrected to 'nuoyafangzhou'.

[0201] Keyboard correction can be a rule-based method, that is, based on the previous input, the current input letter, and the adjacent letters, it determines whether the current input letter should be corrected, and what letter should be corrected.

[0202] 2. Segmentation

[0203] The system can segment the corrected input into pinyin, converting the user-input letter sequence into a pinyin sequence. The pinyin segmentation module divides the user's key input sequence into a pinyin sequence for easier processing by the subsequent phonetic-to-character conversion module. Pinyin is the officially mandated phonetic system for Chinese characters and is the most widely used. In pinyin input methods, users input Chinese characters using pinyin.

[0204] The Pinyin segmentation problem can be solved as a traditional word segmentation problem by using some traditional word segmentation algorithms, such as the maximum matching word segmentation algorithm, the word segmentation algorithm based on the Hidden Markov Model, and so on.

[0205] 3. Phonetic-to-character conversion

[0206] You can input a pinyin sequence into the phonetic-to-character conversion module, which will convert the pinyin sequence into words (i.e., candidate words). The phonetic-to-character conversion will convert the pinyin sequence into a Chinese character sequence and finally provide the result to the user.

[0207] Existing solutions model input error correction, pinyin segmentation, and phonetic-to-character conversion separately. This sequential modeling approach is prone to cascading and amplifying errors; that is, errors in earlier tasks can cause errors in later tasks, which, combined with errors in the later tasks themselves, can create even larger errors. For example, errors in keyboard input correction are very likely to cause errors in pinyin segmentation, which in turn can lead to errors in phonetic-to-character conversion.

[0208] To address the aforementioned issues, this application provides a word and sentence generation method.

[0209] Specifically, this application provides an embodiment of a word and phrase generation method, which can be applied to input method systems for multiple languages ​​such as Chinese, Japanese, and Korean. The input method system can be deployed on a terminal device or on a cloud server. When the input method system is deployed on a cloud server, this embodiment is executed by the cloud server, and the cloud server sends the generated target words and phrases to the terminal device for display.

[0210] Reference Figure 8 , Figure 8 This is an illustration of an embodiment of a word and sentence generation method provided in this application. This method can be applied to an execution device, which can be a terminal device such as a mobile phone, tablet, laptop, or smart wearable device. Figure 8 As shown, the word and phrase generation method provided in this application embodiment may include:

[0211] 801. Obtain the target string sequence, wherein the target string is entered by the user in the input method tool.

[0212] In one possible implementation, the number of characters in the target string sequence is less than a threshold, which is a value less than or equal to 128. For example, the threshold can be 64, 70, 80, 90, 100, 128, etc.

[0213] In this embodiment of the application, a string can be understood as a combination of characters, which is a carrier of language information and is used to generate words and sentences; the words and sentences can be a single word or multiple words, and a single character can also be a word.

[0214] The above input scenarios can be input scenarios in multiple languages ​​such as Chinese, Japanese, and Mandarin; the form of the string is different for different languages; taking Chinese as an example, the string can include one or more pinyin, and in this case, the string can also be called a pinyin string, for example, the string can be "nuoyafangzhou".

[0215] Users can input a target string sequence through an input method tool, and then the terminal device can obtain the target string sequence input by the user.

[0216] 802. Based on the target string sequence, a target neural network is used to generate target words and phrases corresponding to the target string sequence, wherein the target neural network includes an encoder and a decoder, the encoder is used to obtain an embedding vector based on the target string sequence, the decoder is used to generate the target words and phrases based on the embedding vector, and the target neural network is trained using training samples, the training samples including string sequences and corresponding words and phrases.

[0217] In existing implementations, strings input from input method tools require preprocessing through error correction and word segmentation modules before being input into the phoneme-to-character conversion model. In this embodiment, the uncorrected and unsegmented string is directly input into the phoneme-to-character conversion model (e.g., the target neural network in this embodiment). During training of the target neural network, noisy strings can be used directly, enabling the target neural network to have error correction capabilities. Furthermore, since the target string sequence is obtained through input from an input method tool, the character length is not very long (less than a threshold). Even without word segmentation, the target neural network can still obtain accurate words and sentences directly from the original string. This solves the problem of the cumulative error impact of the error correction model and word segmentation model on the accuracy of words and sentences in existing technologies, thus improving the accuracy of word and sentence generation.

[0218] Next, we will first introduce the network structure of the target neural network:

[0219] In one possible implementation, the target neural network includes an encoder and a decoder, wherein the encoder or decoder can be one of the following models: LSTM, GRU, SRU, BERT, Roberta, SpanBERT, XLNet, GPT, Nezha, Mass, Bart, MBART, Albert, StructBERT, Ernie, KnowBERT, K-BERT, TinyBERT.

[0220] In one possible implementation, the encoder can be understood as a deep learning network model. There are various network structures for the encoder, and this application does not specifically limit them. Specifically, the network structure of the encoder can adopt the network structure of the encoder part of the Transformer network, or adopt the network structure of a series of other networks obtained from the encoder part of the Transformer network.

[0221] Taking the encoder or decoder as a bart and the sample string as a pinyin string as an example, the input of a standard bart contains three embedding layers: position embedding, segment embedding, and token embedding. Position embedding distinguishes the different positions of the current token in the sequence; segment embedding distinguishes whether the current token is in the first or second sentence of the input, preparing for the subsequent inter-sentence pre-training task; and token embedding represents the semantics of the current token.

[0222] like Figure 9 As shown, Figure 9This diagram illustrates the construction of an embedding vector for the Pinyin Bart model in this application. Compared to the standard Bart, firstly, the semantics of the input tokens in Pinyin Bart differ. The input tokens in the standard Bart are composed of subwords, typically Chinese characters and common short words, numbering around 30,000. Pinyin Bart, however, addresses the key-to-phoneme conversion problem, using only 26 keyboard keys as input tokens. Secondly, unlike the standard Bart, Pinyin Bart lacks segment tokens because it doesn't require pre-training; it's trained directly on the key-to-phoneme conversion task. Thirdly, the maximum input length of the standard Bart is 512 tokens, which can accommodate a typical article, thus handling document-based tasks. In contrast, Pinyin Bart only handles key-to-phoneme conversion tasks within input methods. Since user input sequences in input method software are generally short, Pinyin Bart's applicability is limited to shorter input sequences, with a maximum sequence length set at 64 or 32 letters, etc. Considering these three factors, the input layer parameters of Pinyin Bart are significantly smaller than those of the standard Bart model.

[0223] In one possible implementation, the encoder can obtain an embedding vector from the target string sequence, where the encoder can process each character in the target string sequence to obtain an embedding vector (or latent vector) for each character. It should be understood that the size of the encoder's input and output can be kept consistent.

[0224] In one possible implementation, the decoder can generate the target phrase based on the embedding vector. Optionally, the decoder can obtain at least one word unit and the probability of each word unit based on the embedding vector, and combine this with a planning algorithm to obtain the target phrase. The planning algorithm can be an N-gram model, the Viterbi algorithm, etc., and is not limited here.

[0225] In one possible implementation, the decoder can sequentially generate word units of the target word phrase based on the embedding vector; that is, the word units generated earlier are used (or described as being used as input) in the subsequent word unit generation. Figure 10 After the target string is encoded by the encoder, the latent vector can be given to the decoder (e.g., encoding the input 'A-E' and then giving it to the decoder). At the same time, on the decoder side, the input sequence is input token by token (e.g., input 'B'), and the expected result is generated one by one based on the input token and the latent vector given by the encoder (e.g., generating 'C').

[0226] In one possible implementation, the decoder can employ a non-autoregressive parallel decoding approach. First, in the key-phoneme conversion problem, the input is a sequence of letters, and the output is a sequence of Chinese characters. Typically, one Chinese character needs to be represented by multiple letters, so the length of the output Chinese character sequence is usually much shorter than the input letter sequence. Therefore, a 'generated sequence length prediction' module is added to the encoder to guide the length of the generated sequence. Second, the decoder is changed from unidirectional Attention (e.g., the GPT model) to bidirectional Attention using the BERT model to support parallel decoding.

[0227] In one possible implementation, the target phrase may include a first character unit and a second character unit, wherein the first character unit is positioned earlier in the target phrase than the second character unit, and the decoder is specifically used to generate the second character unit based on the target string sequence without relying on the fact that the first character unit has already been generated.

[0228] In one possible implementation, the decoder is specifically used to: generate the first word unit and the second word unit in parallel based on the target string sequence.

[0229] In this embodiment of the application, non-autoregressive decoding can significantly improve the inference speed of the model while maintaining the model's performance without a significant decrease.

[0230] Taking the Bart model as an example, through experiments, the differences in inference speed and input performance of the Bart model on autoregressive decoding methods and non-autoregressive decoding methods are compared. The results can be found in Table 1.

[0231] Table 1

[0232]

[0233] The standard Bart model uses an autoregressive decoding method, while the Pinyin Bart uses a non-autoregressive decoding method to improve inference speed. In the table above, the Pinyin Bart constructed using the autoregressive decoding module is denoted as 'Pinyin Bart-AR', where 'AR' stands for 'auto-regressive'.

[0234] In terms of accuracy, when noise is low (1%), the performance of 'Pinyin Bart' using non-autoregressive decoding and 'Pinyin Bart-AR' using autoregressive decoding is similar, with the former experiencing only a 0.03% performance loss. However, in terms of inference speed, the former infers each token (Chinese character) in 1.60ms, while the latter infers in 15.66ms, making the former 9.78 times faster. When noise is high (5%), the accuracy of 'Pinyin Bart' drops further to 0.91%, but still remains below 1%, and the inference speed is still more than 9 times faster (9.30 times). In practical use, this performance drop of less than 1% is generally acceptable, but the resulting speed improvement is significant—making it possible to deploy the model on computing-constrained devices (such as mobile phones and in-vehicle systems), far outweighing the loss from the accuracy decrease. Even with a slight performance loss, the model's accuracy still significantly surpasses that of existing input method engines in this realistic input environment (1% noise: 94.83% vs 66.87%; 5% noise: 91.58% vs 37.75%).

[0235] In one possible implementation, the number of character units in the target word / phrase can be predicted using a character count prediction model based on the target string sequence; an initial word / phrase corresponding to the target string sequence can be generated using a target neural network based on the target string sequence; and the initial word / phrase can be truncated based on the number of character units to obtain the target word / phrase.

[0236] Reference Figure 11 Specifically, after receiving the target string sequence, the target neural network can encode the input sequence through an encoder; predict the length of the target words (number of sub-units) through a word count prediction model; the decoder can generate the initial words corresponding to the target string sequence in parallel based on the encoder's encoding results; finally, the initial words are adjusted based on the previously predicted number of sub-units (e.g., truncating parts that exceed the length).

[0237] In one possible implementation, the word count prediction model can be either a classification model or a regression model.

[0238] In one possible implementation, to improve the accuracy of the target phrase, a word count prediction model can be used to predict the number of sub-units of the target phrase, and the initial phrase can be adjusted based on the number of sub-units.

[0239] In a possible implementation, since users are prone to "false touch phenomena" when inputting characters, that is, the user intends to press a certain key but actually presses another key, in the embodiments of the present application, when the error correction model is removed, in order to ensure that the generated target word or phrase is still accurate when the target string sequence is an input made by the user with an accidental touch, training samples added with noise can be used when training the target neural network. The so-called noise samples refer to string sequences obtained by modifying a correct string sequence (for example, adding characters, deleting characters, or modifying target characters), and the word or phrase corresponding to the string sequence before adding noise is used as a label to form a noise training sample.

[0240] For example, the correct string is "nuoyafangzhou", the string with noise added is "nuiyafangzou", and the word or phrase corresponding to the string sequence before adding noise is "诺亚方舟 (Noah's Ark)". Furthermore, "nuiyafangzou" and "诺亚方舟 (Noah's Ark)" can form a noise training sample.

[0241] Training the target neural network with the above noise samples can enable the target neural network to have error correction capability (that is, for a string containing noise, the target neural network can still generate a correct word or phrase).

[0242] The training process of the target neural network and the construction method of noise samples will be described in subsequent embodiments, and will not be repeated here.

[0243] In a possible implementation, the target string sequence is a string sequence containing noise, the noise is caused by the user's wrong input in the input method tool, and the target word or phrase is the correct word or phrase corresponding to the denoised target string sequence.

[0244] It should be understood that the term "denoised" here does not limit that the target neural network has performed a denoising operation, but means that from the effect of the finally generated target word or phrase, the target word or phrase corresponds to the denoised target string sequence.

[0245] 803. Presenting the target word or phrase on the interface of the input method tool.

[0246] In a possible implementation, the target word or phrase may be displayed as a candidate word on the interface of the input method tool, for example, it may be prompted as the first-priority word or phrase, where the first-priority word or phrase is the word or phrase ranked first among multiple words or phrases prompted by the input method.

[0247] For example, it can be presented by Figure 2 or Figure 3 to present the target word or phrase on the interface of the input method tool.

[0248] Next, taking the target neural network as the pinyin Bart as an example, the effects of the embodiments of this application will be introduced in conjunction with experimental data.

[0249] Table 2

[0250] Existing engines none none 84.56％ NA Existing engines none have 79.30％ 5.26％↓ Existing engines Yes (1%) have 66.87％ 17.69％↓ Existing engines Yes (5%) have 37.75％ 46.81％↓ Pinyin Bert none none 95.59％ 11.03％↑ Pinyin Bert none have 92.22％ 3.72％↓ Pinyin Bert Yes (1%) have 82.77％ 12.82↓ Pinyin Bert Yes (5%) have 56.24％ 39.35↓ Pinyin Bart Yes (1%) have 94.86％ 27.99％ / 12.09％↑ Pinyin Bart Yes (5%) have 92.49％ 54.74％ / 36.25％↑

[0251] 'Existing Engine' refers to a replica of the commonly used Bigram language model as the engine; 'Pinyin Bert' is an existing Pinyin input method engine using the Bert model architecture; and 'Pinyin Bart' is the input method engine corresponding to the embodiment of this application. Each column in the table represents the model's performance under different levels of noise. 'Input Noise' refers to noise generated by the user during keyboard input, such as 'woainizhongguo' being incorrectly entered as 'woaonizongguo'. Different proportions of noise in the test set result in different model performances. 'Segmentation Noise' refers to the noise introduced during Pinyin segmentation. This experiment uses the maximum matching segmentation method to segment the Pinyin sequence; the noise inherent in this algorithm is Pinyin segmentation noise. 'Accuracy' in the table refers to the accuracy based on 'characters', that is, the number of correct characters provided by the input method for every 100 Chinese characters entered by the user.

[0252] Looking at the table above, firstly, the 'existing engine' achieves an accuracy of 84.56% when there is no input noise or segmentation noise. Adding segmentation noise reduces the accuracy by 5.26%. Further increasing input noise further reduces it by 17.69%. Increasing the noise ratio causes a significant performance drop of 46.81%. This demonstrates that the existing input method engine is highly sensitive to noise, and real-world input noise can drastically reduce its performance.

[0253] Next, let's look at the 'Pinyin Bert' engine. When there is no noise, the Pinyin Bert engine achieves an accuracy of 95.59%, an improvement of 11.03% compared to the 84.56% of the 'existing engines', indicating that the previously proposed Pinyin Bert engine performs significantly better than existing input method engines. When segmentation noise is added, the accuracy drops to 92.22%, a decrease of 3.72%; when input noise is added, the accuracy further decreases to 82.77%, a decrease of 12.82%, slightly less than the decrease in the 'existing engines'; when the noise ratio is increased, the accuracy drops significantly to 56%, a decrease of 39.35%. This shows that although the 'Pinyin Bert' model has strong descriptive capabilities, its performance is significantly reduced in noisy environments.

[0254] Finally, the experimental results of the 'Pinyin Bart' engine proposed in this application embodiment are presented. Because this application embodiment unifies the keystroke correction process, pinyin segmentation process, and phonetic-character conversion process into a single process, there is no situation without noise. The experimental process naturally includes user keyboard input noise and pinyin segmentation noise. Under these two noise environments, 'Pinyin Bart' achieved an accuracy rate of 94.68%, an improvement of 27.99% compared to the 'existing engine' under the same environment, and an improvement of 12.09% compared to the 'Pinyin Bert' engine under the same environment. When the noise ratio is increased, the performance of 'Pinyin Bart' decreases from 94.68% to 92.49%, a decrease of only 2.19%, far less than the decrease of the 'existing engine' and 'Pinyin Bert'. Compared to the 'existing engine' under the same conditions, the performance is improved by 54.74%, and compared to 'Pinyin Bert' under the same environment, the performance is improved by 36.25%, a very significant improvement. The above experiments show that the method and Pinyin Bart model proposed in this application can greatly improve the performance of the input method in noisy environments.

[0255] This application provides a word / phrase generation method, comprising: acquiring a target string sequence, wherein the target string is input by a user in an input method tool; generating target words / phrases corresponding to the target string sequence through a target neural network, wherein the target neural network includes an encoder and a decoder, the encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target words / phrases based on the embedding vector, the target neural network is trained using training samples, wherein the training samples include string sequences and corresponding words / phrases; and presenting the target words / phrases in the interface of the input method tool. By inputting uncorrected and unsegmented strings into a phonetic-to-character conversion model (such as the target neural network in this embodiment), the method solves the problem of the cumulative error impact on word / phrase accuracy caused by the error correction model and the segmentation model in the prior art, thereby improving the accuracy of word / phrase generation.

[0256] Reference Figure 12 , Figure 12 This application provides a flowchart illustrating a sample construction method, as shown in the embodiments below. Figure 12 As shown, the sample construction method provided in this application embodiment includes:

[0257] 1201. Obtain the first string sequence and the corresponding words / phrases, wherein the first string sequence includes the first character;

[0258] In a possible implementation, the first character string sequence may be a character string before adding noise. For example, the first character string sequence may be 'woainizhongguo', and its corresponding correct word is 'wo ai ni zhong guo'

[0259] When constructing a sample, since character string sequences are difficult to obtain, in order to obtain the first character string sequence, the words can be converted into the first character string sequence through a pronunciation conversion module. The pronunciation conversion module converts a Chinese character sequence, such as 'wo ai ni zhong guo', into a pinyin sequence, such as 'wo ai ni zhong guo', and then combines it into 'woainizhongguo'. The idea of the pronunciation conversion algorithm is generally to segment Chinese corpora first, then mark the corpora with pinyin according to the pinyin corresponding to the words.

[0260] In a possible implementation, the first character is a character in the first character string sequence.

[0261] In a possible implementation, the first character may be obtained by random sampling (or other methods) on the characters of the first character string sequence. The first character can be used as an object for adding noise to the first character string sequence (specifically, it can be replacing the first character with another character other than the first character, or adding other characters other than the first character before or after the first character).

[0262] In a possible implementation, the first character string sequence can be traversed, and three operations are randomly performed at a certain ratio (for example, 1%): adding a letter at the current traversed position (the position where the first character is located), deleting a letter, and replacing a letter.

[0263] 1202. Determine, by means of a target probability model, the target character corresponding to the first character from at least one second character, wherein the target probability model represents the probability that a user mistakenly touches the virtual key corresponding to each of the at least one second character when the user inputs the first character on a virtual keyboard, and the probability is related to at least one of the following: size information of the virtual key, layout information of the virtual key, the user's operating habits or the user's hand structure characteristics.

[0264] Wherein, the target probability model can be used to describe the probability that the user mistakenly touches the virtual key corresponding to each of the at least one second character when the user inputs the first character on the virtual keyboard.

[0265] In a possible implementation, when a user makes a false touch, the probabilities of different mistakenly touched keys may not be equal. It may be related to information such as size information of the virtual keys, layout information of the virtual keys, the user's operating habits or the user's hand structure characteristics.

[0266] In one possible implementation, the larger the size of the virtual button, the greater the probability of it being accidentally pressed. For example, when a user presses button A, and there are buttons B, C, and D nearby, if the size of button B is larger than the sizes of buttons C and D, then the probability of the user accidentally pressing button B while pressing button A is higher.

[0267] In one possible implementation, keyboards with different virtual key size information can correspond to different target probability models.

[0268] In one possible implementation, the layout information of the virtual keys may include the arrangement of the keys on the keyboard, the distance between the keys, and the shape of the keys themselves. For example, when a user presses key A, if key A is near keys B, C, and D, and the distance between key B and key A is less than the distance between key C and key D and key A, then the probability of the user accidentally pressing key B when pressing key A is higher.

[0269] In one possible implementation, a keyboard with layout information for different virtual keys can correspond to different target probability models.

[0270] In one possible implementation, user operating habits can be understood as the user's habitual actions when pressing keys. Different users may have different habits. For example, if key A is near keys B, C, and D, user A is more likely to accidentally press key B when pressing key A due to their operating habits. Thus, the probability of user A accidentally pressing key B when pressing key A is higher. Another example is that operating habits may be related to keyboard input proficiency.

[0271] In one possible implementation, users with different operating habits can correspond to different target probability models.

[0272] In one possible implementation, a user's hand structure features can be understood as the structural features of the fingers when pressing keys, such as the size of the area between the fingers and the contact surface. Hand structure features can be age-related; even for users of the same age, different hand structure features may correspond to different genders and individual differences.

[0273] In one possible implementation, users with different hand structure features can correspond to different target probability models.

[0274] In one possible implementation, the size information of the virtual button may include the size information of at least one second character.

[0275] In one possible implementation, the layout information of the virtual key may include at least one layout feature between the second character and the first character.

[0276] In one possible implementation, the target probability model can be constructed by pre-collecting user case click behavior. For example, refer to... Figure 13 , Figure 13 This is a distribution feature map of the point cloud of pressure points when a user clicks a key. It can be seen that the area and range of each key click are different during actual input.

[0277] In one possible implementation, the point cloud of the pressure points (or click point cloud, click position point cloud, etc.) when the sample user actually inputs characters on the virtual keyboard can be obtained. The point cloud of pressure points can characterize the user's operating habits, and the distribution of each pressure point can also be related to the size and layout of the keyboard itself as well as the user's hand characteristics.

[0278] In one possible implementation, a target probability model can be constructed based on the aforementioned press point cloud. For example, a corresponding target probability model can be constructed for each character. Each target probability model can represent the probability that the user accidentally touches other virtual keys when inputting the character corresponding to the target probability model.

[0279] For each button's click location, modeling can be performed to construct a target probability model (e.g., Gaussian modeling can be used to construct a Gaussian model). Specifically, for each button, the mean and variance of the click locations are calculated, which can be described by the following Gaussian distribution formula:

[0280]

[0281] Here, σ can represent variance. The smaller the variance, the more stable the data set; the larger the variance, the less stable the data set. Optionally, variance is equal to the average of the squared deviations of each data point (e.g., the distance from the press point to the center of the button) from its arithmetic mean.

[0282] Here, μ can represent the mean. Optionally, the mean is equal to the average of all data points (e.g., the distance from the press point to the center of the button).

[0283] Based on the above formula, the probability that any point on the keyboard is intended to input the current letter can be calculated.

[0284] By using the target probability model, we can obtain the probability that a user will accidentally touch other characters (at least one second character) when inputting the first character. Based on this probability, we can sample at least one second character (or in other ways) to determine the target character to replace the first character from at least one second character.

[0285] For example: if the input intention of the current user is the letter 's', the user may actually accidentally touch letters such as 'a', 'd', 'z', etc. This probability can be obtained through the following steps: first obtain the coordinates of the center position of the accidentally touched letter on the keyboard, for example, the coordinates of the center point of the key of the letter 'a'. Then, according to these coordinates, combined with the Gaussian model of the letter 's', calculate the probability that the input intention is 's' but the user accidentally touches 'a'.

[0286] It should be understood that the character string in the embodiments of the present application may include at least one character (for example, characters corresponding to virtual keys such as English letters, punctuation marks, etc.).

[0287] In the embodiments of the present application, determining the target character for replacing the first character through the target probability model can more accurately characterize the actual behavior of the user, that is, determine the character that is more likely to be accidentally touched. Consequently, the noisy training samples obtained can better reflect the actual user operation situation, and further, the target neural network trained based on the noisy training samples is also more accurate, thereby enhancing the robustness of the model in real user input scenarios.

[0288] 1203. Replace the first character in the first character string sequence with the target character, or add the first character before or after the target character to obtain a second character string sequence, wherein the second character string sequence and the words and phrases are used as training samples of a target neural network, and the target neural network is configured to generate corresponding words and phrases according to the character string sequence.

[0289] After determining the target character for replacing the first character, the first character in the first character string sequence may be replaced with the target character, or the first character may be added before or after the target character to obtain a second character string sequence, wherein the second character string sequence and the words and phrases are used as training samples of a target neural network, and the target neural network is configured to generate corresponding words and phrases according to the character string sequence.

[0290] Refer to Figure 14 , for example, the pinyin sequence 'woainizhongguo' can be noised into 'woaonizongguo' (replacing i with o), which together with the corresponding Chinese character sequence 'wo ai ni zhong guo' (I love you China) forms the noisy training corpus.

[0291] For example, the pinyin sequence 'woainizhongguo' can be noised into 'woaoinizongguo' (adding o before i), which together with the corresponding Chinese character sequence 'wo ai ni zhong guo' (I love you China) forms the noisy training corpus.

[0292] For example, the pinyin sequence 'woainizhongguo' can be noised into 'woaionizongguo' (adding o after i), which, combined with the corresponding Chinese character sequence 'Wo Ai Ni Zhong Guo' (I love you China), together forms the noisy training corpus.

[0293] In a possible implementation, after obtaining training samples, the target neural network can be trained through the training samples, and the target neural network can be the network described in the foregoing embodiment.

[0294] Specifically, the target neural network can be trained according to the second character string sequence and the correct words.

[0295] Next, the process of training the target neural network is described:

[0296] Through the above method, a large number of training samples including noise can be obtained, so training can be performed directly on the target neural network without pre-training.

[0297] In addition, optionally, the word number prediction module can also be used to predict the number of sub-units of a phrase, therefore, in addition to the training loss of the target neural network, the training loss of the word number prediction module can also be constructed.

[0298] Exemplarily, the pinyin-to-character conversion task implemented by the target neural network is a standard sequence token classification task, so cross entropy loss can be used as the loss function, and the cross entropy loss can be in the following form:

[0299]

[0300] wherein is the predicted value of the i-th token in the output sequence by the model, y i is the true value.

[0301] For the word number prediction module, there are also different loss selections and design methods according to different problem conversion methods. First, length prediction can also be converted into a classification problem, for example: assuming that the maximum output length of the model is 64, then the output of the length prediction module is converted into a classification problem of 1 to 64 categories. This can also be described by cross entropy loss. In addition, the length prediction problem can also be converted into a regression problem, that is, the model predicts a real number to represent the length. In this case, mean squared error (MSE) is used for description, and the loss can be in the following form:

[0302]

[0303] In one possible implementation, the overall loss of the model during training consists of the two losses mentioned above, which can be calculated using a weighted average, for example, in the following form:

[0304] Loss total =w1*Loss mse +w2*Loss cross entropy ;

[0305] The weights w1 and w2 can be manually specified based on experience. For the loss fusion method, other more complex methods that are currently available can also be used, such as the GradNorm method.

[0306] The beneficial effects of the embodiments of this application will be described below with reference to experimental results. The experimental results are shown in the table below:

[0307] Table 3

[0308] Bart (Uniform Noise) Yes (1%) have 92.57％ --- Bart (Uniform Noise) Yes (5%) have 87.00％ --- Pinyin Bart (real noise) Yes (1%) have 94.86％ 2.29％↑ Pinyin Bart (real noise) Yes (5%) have 92.49％ 5.49％↑

[0309] The experimental data in the table above shows that the Pinyin Bart model trained with uniformly noisy training corpus has a lower accuracy than the model trained with real user input noise. Furthermore, the performance degradation becomes more pronounced as the noise level increases. These results demonstrate that the present invention generates a large-scale training corpus with noise, which can fully reflect the noise conditions of real user input scenarios, and the model trained on this basis has stronger robustness.

[0310] This application provides a sample construction method, the method comprising: acquiring a first string sequence and corresponding phrases, the first string sequence including a first character; determining a target character corresponding to the first character from at least one second character using a target probability model, wherein the target probability model represents the probability that a user accidentally touches a virtual key corresponding to each of the at least one second character when inputting the first character on a virtual keyboard, the probability being related to at least one of the following: the size information of the virtual key, the layout information of the virtual key, the user's operating habits, or the user's hand structure features; replacing the first character in the first string sequence with the target character, or adding the first character before or after the target character to obtain a second string sequence, the second string sequence and the phrases being used as training samples for a target neural network, the target neural network being used to generate corresponding phrases based on the string sequence. In this embodiment, the target character used to replace the first character is determined by the target probability model, which can more accurately depict the user's actual behavior, that is, determine the character that is more likely to be accidentally touched. As a result, the noise-added training samples obtained can better reflect the actual user operation. Consequently, the target neural network trained based on the noise-added training samples is more accurate, thereby enhancing the robustness of the model in real user input scenarios.

[0311] Reference Figure 15 , Figure 15 This application provides a schematic diagram of the structure of a word and phrase generation device, as shown in the embodiments. Figure 15 As shown, the word and sentence generation apparatus 1500 provided in this application includes:

[0312] The acquisition module 1501 is used to acquire a target string sequence, wherein the target string is input by the user in the input method tool;

[0313] The description of the acquisition module 1501 can be found in the description of step 801 in the above embodiments, and will not be repeated here.

[0314] The word / phrase generation module 1502 is used to generate target words / phrase corresponding to the target string sequence through a target neural network based on the target string sequence. The target neural network includes an encoder and a decoder. The encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target words / phrase based on the embedding vector. The target neural network is trained through training samples, and the training samples include string sequences and corresponding words / phrase.

[0315] The description of the word and sentence generation module 1502 can be found in the description of step 802 in the above embodiment, and will not be repeated here.

[0316] The presentation module 1503 is used to present the target words and phrases in the interface of the input method tool.

[0317] The description of the presentation module 1503 can be found in the description of step 803 in the above embodiments, and will not be repeated here.

[0318] In one possible implementation, the number of characters in the target string sequence is less than a threshold, which is a value less than or equal to 128.

[0319] In one possible implementation, the target phrase includes a first character unit and a second character unit, wherein the first character unit is positioned earlier in the target phrase than the second character unit, and the decoder is specifically used to generate the second character unit based on the target string sequence without relying on the fact that the first character unit has already been generated.

[0320] In one possible implementation, the decoder is specifically used to: generate the first word unit and the second word unit in parallel based on the target string sequence.

[0321] In one possible implementation, the device further includes:

[0322] The word count prediction module is used to predict the number of character units of the target word / sentence based on the target string sequence using a word count prediction model.

[0323] The word and sentence generation module is specifically used for:

[0324] Based on the target string sequence, an initial phrase corresponding to the target string sequence is generated using a target neural network;

[0325] Based on the number of character units, the initial word / phrase is truncated to obtain the target word / phrase.

[0326] In one possible implementation, the target string sequence is a string sequence containing noise, the noise being caused by user mis-input in the input method tool;

[0327] The target phrase is the correct phrase corresponding to the target string sequence after noise removal.

[0328] In one possible implementation, the encoder or decoder is one of the following models:

[0329] LSTM, GRU, SRU, bert, roberta, spanbert, xlnet, GPT, nezha, mass, bart, mbart, albert, structbert, ernie, knowbert, k-bert, tinybert.

[0330] Reference Figure 16 , Figure 16 This is a schematic diagram of the structure of a sample construction device provided in an embodiment of this application, such as... Figure 16 As shown, the sample construction apparatus 1600 provided in this application embodiment may include:

[0331] The acquisition module 1601 is used to acquire a first string sequence and the corresponding words and phrases, wherein the first string sequence includes a first character;

[0332] The description of the acquisition module 1601 can be found in the description of step 1201 in the above embodiments, and will not be repeated here.

[0333] The character replacement module 1602 is used to determine the target character corresponding to the first character from at least one second character using a target probability model, wherein the target probability model represents the probability that a user accidentally touches a virtual key corresponding to each of the at least one second character when inputting the first character on a virtual keyboard, and the probability is related to at least one of the following:

[0334] The size information of the virtual buttons, the layout information of the virtual buttons, the user's operating habits, or the user's hand structure characteristics;

[0335] The first character in the first string sequence is replaced with the target character, or the first character is added before or after the target character to obtain a second string sequence. The second string sequence and the phrase are used as training samples for the target neural network, which is used to generate corresponding phrases based on the string sequence.

[0336] The description of the character replacement module 1602 can be found in the descriptions of steps 1202 and 1203 in the above embodiments, and will not be repeated here.

[0337] In one possible implementation, the target probability model is a Gaussian probability model.

[0338] In one possible implementation, the device further includes:

[0339] The training module is used to train the target neural network based on the second string sequence and the correct words / phrases.

[0340] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 17 , Figure 17 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1700 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, or a server, etc., and is not limited thereto. Specifically, the execution device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703, and a memory 1704 (wherein the execution device 1700 may have one or more processors 1703). Figure 17 (Taking a processor as an example), processor 1703 may include application processor 17031 and communication processor 17032. In some embodiments of this application, receiver 1701, transmitter 1702, processor 1703 and memory 1704 may be connected via a bus or other means.

[0341] Memory 1704 may include read-only memory and random access memory, and provides instructions and data to processor 1703. A portion of memory 1704 may also include non-volatile random access memory (NVRAM). Memory 1704 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0342] Processor 1703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0343] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1703. Processor 1703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1703 or by instructions in software form. Processor 1703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1703 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1704. Processor 1703 reads the information in memory 1704 and, in conjunction with its hardware, completes the steps of the above method.

[0344] Receiver 1701 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1702 can be used to output digital or character information; transmitter 1702 can also be used to send instructions to the disk group to modify the data in the disk group.

[0345] In one embodiment of this application, the processor 1703 is used to execute the word generation method and sample construction method executed by the execution device in the above embodiments (e.g., the step of performing model inference through the target neural network).

[0346] This application also provides a training device; please refer to [link / reference]. Figure 18 , Figure 18This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 1800 is implemented by one or more servers. The training device 1800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1818 (e.g., one or more processors) and memory 1832, and one or more storage media 1830 (e.g., one or more mass storage devices) for storing application programs 1842 or data 1844. The memory 1832 and storage media 1830 can be temporary or persistent storage. The program stored in the storage media 1830 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1818 may be configured to communicate with the storage media 1830 and execute the series of instruction operations in the storage media 1830 on the training device 1800.

[0347] The training device 1800 may also include one or more power supplies 1826, one or more wired or wireless network interfaces 1850, one or more input / output interfaces 1858; or, one or more operating systems 1841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0348] In this embodiment, the central processing unit 1818 is used to execute the steps related to model training in the above embodiments.

[0349] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0350] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0351] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0352] For details, please refer to Figure 19 , Figure 19 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1900. The NPU 1900 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1903, which is controlled by the controller 1904 to extract matrix data from the memory and perform multiplication operations.

[0353] In some implementations, the arithmetic circuit 1903 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1903 is a two-dimensional pulsating array. The arithmetic circuit 1903 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1903 is a general-purpose matrix processor.

[0354] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1902 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1901 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1908.

[0355] Unified memory 1906 is used to store input and output data. Weight data is directly transferred to weight memory 1902 via Direct Memory Access Controller (DMAC) 1905. Input data is also transferred to unified memory 1906 via DMAC.

[0356] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1909.

[0357] The Bus Interface Unit (BIU) 1910 is used by the instruction fetch memory 1909 to fetch instructions from external memory, and also by the memory access controller 1905 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0358] The DMAC is mainly used to move input data from external memory DDR to unified memory 1906, or to weight data to weight memory 1902, or to input data to input memory 1901.

[0359] The vector computation unit 1907 includes multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0360] In some implementations, the vector computation unit 1907 can store the processed output vector in the unified memory 1906. For example, the vector computation unit 1907 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1903, such as linear interpolation of feature planes extracted by a convolutional layer, or, for example, a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1907 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1903, for example, for use in subsequent layers of the neural network.

[0361] The instruction fetch buffer 1909 connected to the controller 1904 is used to store the instructions used by the controller 1904;

[0362] Unified memory 1906, input memory 1901, weighted memory 1902, and instruction fetch memory 1909 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0363] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0364] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0365] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0366] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0367] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for generating words and sentences, characterized in that, The method includes: Obtain the target string sequence, wherein the target string is entered by the user in the input method tool; Based on the target string sequence, a target neural network is used to generate target words and phrases corresponding to the target string sequence. The target neural network includes an encoder and a decoder. The encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target words and phrases based on the embedding vector. The target neural network is trained using training samples, which include string sequences and corresponding words and phrases. The target words and phrases are displayed in the interface of the input method tool; The method further includes: Based on the target string sequence, the number of character units of the target word / sentence is predicted using a character count prediction model; The step of generating target words and phrases corresponding to the target string sequence through a target neural network includes: Based on the target string sequence, an initial phrase corresponding to the target string sequence is generated using a target neural network; Based on the number of character units, the initial word / phrase is truncated to obtain the target word / phrase.

2. The method according to claim 1, characterized in that, The number of characters in the target string sequence is less than a threshold, which is a value less than or equal to 128.

3. The method according to claim 1, characterized in that, The target phrase includes a first character unit and a second character unit. The first character unit is positioned earlier in the target phrase than the second character unit. The decoder is specifically used to generate the second character unit based on the target string sequence without relying on the first character unit having already been generated.

4. The method according to claim 3, characterized in that, The decoder is specifically used to: generate the first word unit and the second word unit in parallel according to the target string sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The target string sequence is a string sequence containing noise, which is caused by the user's erroneous input in the input method tool; The target phrase is the correct phrase corresponding to the target string sequence after noise removal.

6. The method according to claim 1, characterized in that, The encoder or decoder is one of the following models: LSTM, GRU, SRU, bert, roberta, spanbert, xlnet, GPT, nezha, mass, bart, mbart, albert, structbert, ernie, knowbert, k-bert, tinybert.

7. A sample construction method, characterized in that, The method includes: Obtain the first string sequence and its corresponding words, wherein the first string sequence includes the first character; A target probability model is used to determine the target character corresponding to the first character from at least one second character. The target probability model represents the probability that a user accidentally presses a virtual key corresponding to each of the at least one second character when inputting the first character on a virtual keyboard. This probability is related to at least one of the following: The size information of the virtual buttons, the layout information of the virtual buttons, the user's operating habits, or the user's hand structure characteristics; The first character in the first string sequence is replaced with the target character, or the first character is added before or after the target character to obtain a second string sequence. The second string sequence and the phrase are used as training samples for the target neural network. The target neural network is used to generate corresponding phrases based on the string sequence. The target neural network is used in the method of claim 1.

8. The method according to claim 7, characterized in that, The target probability model is a Gaussian probability model.

9. The method according to claim 7 or 8, characterized in that, The method further includes: Obtain the dot cloud of key presses on the virtual keyboard when a sample user inputs characters; Based on the point cloud of the pressed points, the target probability model is constructed.

10. A word / sentence generation device, characterized in that, The device includes: The acquisition module is used to acquire a target string sequence, wherein the target string is entered by the user in the input method tool; A word / phrase generation module is used to generate target words / phrase corresponding to the target string sequence through a target neural network based on the target string sequence. The target neural network includes an encoder and a decoder. The encoder is used to obtain an embedding vector based on the target string sequence, and the decoder is used to generate the target words / phrase based on the embedding vector. The target neural network is trained using training samples, which include string sequences and corresponding words / phrase. The presentation module is used to present the target words and phrases in the interface of the input method tool; The device further includes: The word count prediction module is used to predict the number of character units of the target word / sentence based on the target string sequence using a word count prediction model. The word and sentence generation module is specifically used for: Based on the target string sequence, an initial phrase corresponding to the target string sequence is generated using a target neural network; Based on the number of character units, the initial word / phrase is truncated to obtain the target word / phrase.

11. The apparatus according to claim 10, characterized in that, The number of characters in the target string sequence is less than a threshold, which is a value less than or equal to 128.

12. The apparatus according to claim 10, characterized in that, The target phrase includes a first character unit and a second character unit. The first character unit is positioned earlier in the target phrase than the second character unit. The decoder is specifically used to generate the second character unit based on the target string sequence without relying on the first character unit having already been generated.

13. The apparatus according to claim 12, characterized in that, The decoder is specifically used to: generate the first word unit and the second word unit in parallel according to the target string sequence.

14. The apparatus according to any one of claims 10 to 13, characterized in that, The target string sequence is a string sequence containing noise, which is caused by the user's erroneous input in the input method tool; The target phrase is the correct phrase corresponding to the target string sequence after noise removal.

15. The apparatus according to claim 10, characterized in that, The encoder or decoder is one of the following models: LSTM, GRU, SRU, bert, roberta, spanbert, xlnet, GPT, nezha, mass, bart, mbart, albert, structbert, ernie, knowbert, k-bert, tinybert.

16. A sample construction apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first string sequence and its corresponding words and phrases, wherein the first string sequence includes a first character; A character replacement module is used to determine the target character corresponding to the first character from at least one second character using a target probability model, wherein the target probability model represents the probability that a user accidentally presses a virtual key corresponding to each of the at least one second character when inputting the first character on a virtual keyboard, and the probability is related to at least one of the following: The size information of the virtual buttons, the layout information of the virtual buttons, the user's operating habits, or the user's hand structure characteristics; The first character in the first string sequence is replaced with the target character, or the first character is added before or after the target character to obtain a second string sequence. The second string sequence and the phrase are used as training samples for the target neural network, which is used to generate corresponding phrases based on the string sequence. The target neural network is used in the apparatus of claim 10.

17. The apparatus according to claim 16, characterized in that, The target probability model is a Gaussian probability model.

18. The apparatus according to claim 16 or 17, characterized in that, The acquisition module is also used for: Obtain the dot cloud of key presses on the virtual keyboard when a sample user inputs characters; Based on the point cloud of the pressed points, the target probability model is constructed.

19. A data processing apparatus, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 9.

Images