A method and device for generating motion

Abstract

The present application discloses an action generation method and device in the field of artificial intelligence, which may include: obtaining an input corpus, the input corpus including at least one word; obtaining a word sequence according to the input corpus, the word sequence including a vector corresponding to at least one word; obtaining an action sequence, the action sequence including a vector corresponding to at least one action, the at least one action including an action corresponding to at least one word; fusing the word sequence and the action sequence to obtain a fused sequence; using the fused sequence as the input of an action generation network, and outputting a parameter set, the parameter set including action parameters, and the action parameters are used to render an image including the action.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an action generation method and device. Background Art

[0002] Usually, the meaning of text can be expressed through actions. For example, a sign language digital human can allow users to obtain information about the world through sign language without any obstacles, that is, through a sign language digital human, text or speech is converted into sign language, so that users can understand the meaning expressed by the text or speech through sign language.

[0003] Usually, the text can be converted into sign language word order through pre-set conversion rules or machine translation, and then the final sign language gestures can be obtained through corresponding action splicing; or, the sign language gestures can be directly generated from Chinese text or sign language text through a neural network. However, the sign language gestures generated by these methods are usually incoherent or limited by limited data sets and have poor generalization, resulting in inaccurate meaning and low user experience. Summary of the invention

[0004] The present application provides an action generation method and device for generating actions based on text drive, by inputting fused text and action information into a neural network, making full use of the action database and the generation capability of the neural network to synthesize sign language actions.

[0005] In view of this, in a first aspect, the present application provides an action generation method, comprising: obtaining an input corpus, the input corpus including at least one word; obtaining a word sequence according to the input corpus, the word sequence including a vector corresponding to at least one word; obtaining an action sequence, the action sequence including a vector corresponding to at least one action, the at least one action including an action corresponding to at least one word; fusing the word sequence and the action sequence to obtain a fused sequence; using the fused sequence as an input of an action generation network, outputting a parameter set, the parameter set including action parameters, the action parameters being used to render an action image, and the action generation network being used to convert the input vector into parameters related to the action.

[0006] Therefore, in the implementation mode of the present application, after obtaining the action vector representing the action, the action corresponding to the input corpus can be preliminarily determined. The fusion vector combines the information included in the word vector and the action vector, and then further outputs the action parameters through the neural network, thereby obtaining more accurate and continuous actions and improving the user experience.

[0007] In a possible implementation, the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0008] In the implementation manner of the present application, the action generation network can also output expression parameters for rendering to obtain an image including the expression, so that the user can more accurately know the information included in the input corpus through the expression image, thereby improving the user experience.

[0009] In a possible implementation, the method may further include: obtaining an expression sequence based on an input corpus, the expression sequence including a vector corresponding to at least one expression; fusing the aforementioned word sequence and action sequence to obtain a fused sequence, which may include: fusing a word sequence, an action sequence and an expression sequence to obtain a fused sequence.

[0010] Therefore, in the implementation manner of the present application, an expression sequence can also be obtained based on the input corpus, and then in the process of obtaining the fused sequence, in addition to fusing the word sequence and the action sequence, the expression sequence is also fused, so as to facilitate subsequent rendering to obtain the expression action, so that the user can more accurately know the information included in the input corpus through the expression image, thereby improving the user experience.

[0011] In a possible implementation, before using the fused sequence as the input of the action generation network and outputting a parameter set, the method may further include: training the action generation network using a training set, wherein the training set includes a plurality of sample pairs, each sample pair including a set of word sequences and at least one corresponding set of action parameters.

[0012] Therefore, in the implementation manner of the present application, the samples used to train the neural network may include a whole set of word sequences and corresponding action parameters, so that the actions corresponding to the output parameters of the trained neural network are more coherent, thereby improving the user experience.

[0013] In a possible implementation, the above-mentioned acquisition of word sequences based on input corpus may include: converting the input corpus according to preset rules to obtain a word sequence; or, using the input corpus as input of a word vector conversion network and outputting a word sequence, and the word vector conversion network is used to convert the corpus into corresponding vectors.

[0014] In the implementation manner of the present application, word sequences can be obtained in a variety of ways, thereby being able to adapt to more application scenarios.

[0015] In a possible implementation, the above-mentioned obtaining the action sequence may include: searching a preset action database for actions corresponding to the input corpus to obtain the action sequence.

[0016] In the implementation manner of the present application, an action database can be pre-set. After obtaining the input corpus, the corresponding action can be queried from the action database according to the vocabulary in the input corpus to obtain the action vector, thereby obtaining the action sequence efficiently and accurately.

[0017] In a possible implementation, the above-mentioned obtaining of input corpus may include: extracting text from at least one of text, voice or image data to obtain the input corpus.

[0018] Therefore, in the implementation manner of the present application, input corpus can be obtained through various channels, so that it can be applied in various scenarios to improve user experience.

[0019] In a second aspect, the present application provides a neural network for action generation, the neural network is used to perform the method steps in the aforementioned first aspect or any embodiment of the first aspect, and the neural network may specifically include:

[0020] The text acquisition module is used to acquire input corpus, which includes at least one word.

[0021] A word vector conversion network, used for obtaining a word sequence according to an input corpus, wherein the word sequence includes a vector corresponding to at least one word;

[0022] An action vector conversion network, used to obtain an action sequence, wherein the action sequence includes a vector corresponding to at least one action, and the at least one action includes an action corresponding to at least one word;

[0023] A fusion network is used to fuse word sequences and action sequences to obtain a fused sequence;

[0024] The action generation network takes the fusion sequence as the input of the action generation network and outputs a parameter set, which includes action parameters. The action parameters are used to render the action image. The action generation network is used to convert the input vector into parameters related to the action.

[0025] In a possible implementation, the parameter set also includes expression parameters, and the expression parameters are used to render expression actions corresponding to at least one action.

[0026] In a possible implementation, the neural network may further include:

[0027] An expression vector conversion network is used to obtain an expression sequence according to an input corpus, wherein the expression sequence includes a vector corresponding to at least one expression;

[0028] The fusion network is specifically used to fuse word sequences, action sequences and expression sequences to obtain a fused sequence.

[0029] In a possible implementation, a training set is used to train a neural network, where the training set includes a plurality of sample pairs, and each sample pair includes a set of word sequences and at least one corresponding set of action parameters.

[0030] In a possible implementation, the text acquisition module is specifically configured to extract text from at least one of text, voice or image data to obtain input corpus.

[0031] In a third aspect, an embodiment of the present application provides an action generation device, which has the function of implementing the action generation method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0032] Specifically, the action generating device may include:

[0033] A text acquisition module, used to acquire input corpus, the input corpus includes at least one word;

[0034] A word vector conversion module, used to obtain a word sequence according to an input corpus, wherein the word sequence includes a vector corresponding to at least one word;

[0035] An action vector conversion module, used to obtain an action sequence, wherein the action sequence includes a vector corresponding to at least one action, and the at least one action includes an action corresponding to at least one word;

[0036] A fusion module is used to fuse word sequences and action sequences to obtain a fused sequence;

[0037] The output module is used to use the fusion sequence as the input of the action generation network, and output a parameter set, which includes action parameters. The action parameters are used to render the action image. The action generation network is used to convert the input vector into parameters related to the action.

[0038] In a possible implementation, the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0039] In a possible implementation, the device further includes:

[0040] An expression vector conversion module, used to obtain an expression sequence according to an input corpus, wherein the expression sequence includes a vector corresponding to at least one expression;

[0041] The fusion module is specifically used to fuse word sequences, action sequences and expression sequences to obtain a fused sequence.

[0042] In a possible implementation, the device further includes:

[0043] The training module is used to train the action generation network using a training set before using the fusion sequence as the input of the action generation network and outputting a parameter set. The training set includes multiple sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters.

[0044] In a possible implementation, the word vector conversion module is specifically used to:

[0045] Convert the input corpus or word sequence according to preset rules to obtain a word sequence;

[0046] Alternatively, the input corpus or word sequence is used as the input of the word vector conversion network, and the word vector conversion network is used to convert the corpus into the corresponding vector.

[0047] In one possible implementation, the action vector conversion module is specifically used to query the action corresponding to the input corpus or word sequence from a preset action database to obtain an action sequence; or, use the input corpus or word sequence as the input of the action vector conversion network to output the action sequence.

[0048] In a possible implementation, the text acquisition module is specifically used to extract text from at least one of text, voice or image data, and use the text as input corpus or convert the text to obtain input corpus.

[0049] In a fourth aspect, an embodiment of the present application provides an action generation device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the action generation method shown in any one of the first aspects above. Optionally, the action generation device can be a chip.

[0050] In the fifth aspect, an embodiment of the present application provides an action generating device, which may also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0051] In a sixth aspect, the present application provides a training method, comprising:

[0052] A neural network is trained using a training set to obtain a trained neural network, wherein the training set includes a plurality of sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters, the neural network is used to fuse the input action sequence and word sequence to obtain a fused sequence, and output a parameter set according to the fused sequence, wherein the word sequence includes a vector corresponding to at least one word in the input corpus, the action sequence includes a vector corresponding to at least one action, at least one action includes an action corresponding to at least one word, the parameter set includes action parameters, and the action parameters are used to render an action image.

[0053] Therefore, in the implementation manner of the present application, when training the neural network, each sample pair included in the training set used may include a set of word sequences and corresponding complete action parameters of the action, so that the actions output by the neural network are more coherent, thereby improving the user experience.

[0054] It can be understood that the neural network can be used to execute the steps in the aforementioned first aspect or the optional implementation of the first aspect, or the neural network can be the neural network for action generation provided by the aforementioned second aspect.

[0055] In one possible implementation, a neural network is specifically used to fuse word sequences, action sequences, and expression sequences to obtain a fused sequence, wherein the expression sequence is obtained based on an input corpus, and the expression sequence includes a vector corresponding to at least one expression; the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0056] Therefore, in the implementation manner of the present application, the neural network can also output expression parameters, so that an expression image can be rendered, so that the user can more accurately know the specific information of the input corpus through the expression image, thereby improving the user experience.

[0057] In a possible implementation, before using the training set to train the neural network, the neural network is also pre-trained using a data set to obtain a pre-trained neural network, and the data set may include multiple sample pairs, each sample pair may include a vocabulary or word vector and a corresponding action parameter, and then the training set may be used to fine-tune the pre-trained neural network to obtain a trained neural network.

[0058] Therefore, in the implementation mode of the present application, in the pre-training stage, word vectors and their corresponding action parameters can be used for training, so as to ensure that the final neural network can output accurate actions. In the fine-tuning stage, word sequences and their corresponding complete action parameters can be used for training, so that the final neural network can output more coherent action parameters on the basis of accurate output, thereby improving the user experience.

[0059] In a seventh aspect, an embodiment of the present application provides a training device, which has the function of implementing the action generation method in the sixth aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0060] Specifically, the training device may include:

[0061] A training module is used to train a neural network using a training set to obtain a trained neural network. The training set includes multiple sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters. The neural network is used to fuse the input action sequence and word sequence to obtain a fused sequence, and output a parameter set based on the fused sequence. The word sequence includes a vector corresponding to at least one word in the input corpus, the action sequence includes a vector corresponding to at least one action, at least one action includes an action corresponding to at least one word, and the parameter set includes action parameters, which are used to render an action image.

[0062] In one possible implementation, a neural network is specifically used to fuse word sequences, action sequences, and expression sequences to obtain a fused sequence, wherein the expression sequence is obtained based on an input corpus, and the expression sequence includes a vector corresponding to at least one expression; the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0063] In one possible implementation, the training module is further used to pre-train the neural network using a data set before using the training set to train the neural network to obtain a pre-trained neural network. The data set may include multiple sample pairs, each sample pair may include a vocabulary or word vector and a corresponding action parameter. Subsequently, the training set may be used to fine-tune the pre-trained neural network to obtain a trained neural network.

[0064] In an eighth aspect, an embodiment of the present application provides a training device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the action generation method shown in any one of the first six aspects. Optionally, the training device can be a chip.

[0065] In the ninth aspect, an embodiment of the present application provides a training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing in the sixth aspect or any optional embodiment of the sixth aspect.

[0066] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the sixth aspect.

[0067] In the eleventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the sixth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of an artificial intelligence main framework for this application;

[0069] Figure 2 A schematic diagram of a system architecture provided for this application;

[0070] Figure 3 Another system architecture diagram provided for this application;

[0071] Figure 4 A flowchart of an action generation method provided in an embodiment of the present application;

[0072] Figure 5 A flowchart of another action generation method provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a process of fusing vectors provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of another process of fusing vectors provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of another process of fusing vectors provided in an embodiment of the present application;

[0076] Fig. 9 A schematic diagram of another process of fusing vectors provided in an embodiment of the present application;

[0077] Fig.10 A schematic diagram of a process for outputting parameters by autoregression provided in an embodiment of the present application;

[0078] Fig.11 A schematic diagram of a process for outputting parameters in a non-autoregressive manner provided in an embodiment of the present application;

[0079] Fig.12 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0080] Fig.13 A flowchart of a training method provided in an embodiment of the present application;

[0081] Fig.14 A schematic diagram of a flow chart for constructing training samples provided in an embodiment of the present application;

[0082] Fig.15 A schematic diagram of the structure of an action generation device provided in an embodiment of the present application;

[0083] Fig.16A schematic diagram of the structure of another action generation device provided in an embodiment of the present application;

[0084] Fig.17 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0085] Fig.18 A schematic diagram of the structure of another training device provided in an embodiment of the present application;

[0086] Fig.19 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0088] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0089] (1) Infrastructure

[0090] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, such as central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0091] (2) Data

[0092] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0093] (3) Data processing

[0094] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0095] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0096] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0097] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0098] (4) General ability

[0099] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0100] (5) Smart products and industry applications

[0101] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0102] The embodiments of the present application involve related applications of neural networks and data conversion (natural language processing, NLP). In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0103] Corpus: Also known as free text, it can be words, phrases, sentences, fragments, articles, and any combination thereof. For example, "Today's weather is really nice" is a piece of corpus.

[0104] Loss function: also known as cost function, a measure of the difference between the predicted output of a machine learning model for a sample and the true value of the sample (also known as the supervised value), that is, it is used to measure the difference between the predicted output of a machine learning model for a sample and the true value of the sample. The loss function can usually include mean square error, cross entropy, logarithm, exponential loss functions, etc. For example, the mean square error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0105] Gradient: The vector of derivatives of the loss function with respect to the parameters.

[0106] Stochastic gradient: The number of samples in machine learning is very large, so the loss function calculated each time is calculated using randomly sampled data, and the corresponding gradient is called stochastic gradient.

[0107] Back propagation (BP): An algorithm that calculates the gradient of model parameters based on the loss function and updates the model parameters.

[0108] Neural machine translation: Neural machine translation is a typical task of data conversion. Given a sentence in a source language, the task is to output the corresponding sentence in the target language. In the commonly used neural machine translation model, the words in the sentences of the source language and the target language are encoded into vector representations, and the associations between words and sentences are calculated in the vector space to perform the translation task.

[0109] Pre-trained language model (PLM): a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation to perform downstream tasks, such as the word vector conversion network, action vector conversion network or action generation network mentioned below in this application can all be implemented using PLM. The training of PLM includes two stages, namely the pre-training stage and the fine-tuning stage. In the pre-training stage, the model is trained on language model tasks on large-scale unsupervised text to learn word representation. In the fine-tuning stage, the model is initialized using the parameters learned in the pre-training stage, and is trained in fewer steps on downstream tasks such as text classification or sequence labeling, so that the semantic information obtained from the pre-training can be successfully transferred to downstream tasks.

[0110] The action generation method provided in the embodiment of the present application can be executed on a server or on a terminal device. The terminal device can be a mobile phone with image processing function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a video camera, a smart watch, a wearable device (WD) or an autonomous driving vehicle, etc., and the embodiment of the present application does not limit this.

[0111] See attached Figure 2, the embodiment of the present application provides a system architecture 200. The system architecture includes a database 230 and a client device 240. The data acquisition device 260 is used to collect data and store it in the database 230, and the training module 202 generates a target model / rule 201 based on the data maintained in the database 230. The following will describe in more detail how the training module 202 obtains the target model / rule 201 based on the data. The target model / rule 201 is the neural network mentioned in the following embodiments of the present application. For details, refer to the following Figure 4-Figure 11 Related description in .

[0112] The calculation module may include a training module 202. The target model / rule obtained by the training module 202 may be applied to different systems or devices. Figure 2 In the embodiment, the execution device 210 is configured with a transceiver 212, which may be a wireless transceiver, an optical transceiver or a wired interface (such as an I / O interface), etc., to interact with external devices. A "user" may input data to the transceiver 212 through a client device 240. For example, the client device 240 may send a target task to the execution device 210, request the execution device to train a neural network, and send a database for training to the execution device 210.

[0113] The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0114] The calculation module 211 processes the input data using the target model / rule 201. Specifically, the calculation module 211 is used to: obtain input corpus, which may include one or more words; then obtain a word sequence according to the input corpus, the word sequence includes a vector corresponding to at least one word; obtain an action sequence, the action sequence includes a vector corresponding to at least one action, and at least one action corresponds to at least one word; fuse the word sequence and the action sequence to obtain a fused sequence; obtain a parameter set according to the fused sequence, the parameter set includes action parameters, and the action parameters are used to render at least one action.

[0115] Finally, the transceiver 212 returns the output of the neural network to the client device 240. For example, the user can input a text to be converted into a sign language action through the client device 240, and the neural network outputs the sign language action or the parameters representing the sign language action and feeds back to the client device 240.

[0116] More deeply, the training module 202 can obtain corresponding target models / rules 201 for different tasks based on different data to provide users with better results.

[0117] In the attached Figure 2In the case shown in , the data input into the execution device 210 can be determined based on the user's input data. For example, the user can operate in the interface provided by the transceiver 212. In another case, the client device 240 can automatically input data into the transceiver 212 and obtain the result. If the client device 240 automatically inputs data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also serve as a data collection terminal to store the collected data associated with the target task into the database 230.

[0118] The training or updating process mentioned in the present application can be performed by the training module 202. It is understandable that the training process of the neural network is to learn the way to control the spatial transformation, more specifically, to learn the weight matrix. The purpose of training the neural network is to make the output of the neural network as close to the expected value as possible. Therefore, the weight vector of each layer of the neural network in the neural network can be updated according to the difference between the predicted value and the expected value of the current network (of course, the weight vector can usually be initialized before the first update, that is, the parameters of each layer in the deep neural network are pre-configured). For example, if the predicted value of the network is too high, the value of the weight in the weight matrix is ​​adjusted to reduce the predicted value, and after continuous adjustment, the value output by the neural network is close to or equal to the expected value. Specifically, the difference between the predicted value and the expected value of the neural network can be measured by a loss function or an objective function. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. The training of the neural network can be understood as the process of minimizing the loss as much as possible. The process of updating the weight of the starting network and training the serial network in the following embodiments of the present application can refer to this process, which will not be repeated below.

[0119] like Figure 2As shown, the target model / rule 201 is obtained by training according to the training module 202. The target model / rule 201 can be the self-attention model in the present application in the embodiment of the present application, and the self-attention model can include deep convolutional neural networks (DCNN), recurrent neural networks (RNNS) and the like. The neural network mentioned in the present application can include various types, such as deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN) or residual network other neural networks.

[0120] In the training phase, the database 230 can be used to store sample sets for training. The execution device 210 generates a target model / rule 201 for processing samples, and iteratively trains the target model / rule 201 using the sample set in the database to obtain a mature target model / rule 201, which is specifically represented by a neural network. The neural network obtained by the execution device 210 can be applied to different systems or devices.

[0121] In the inference stage, the execution device 210 can call the data, code, etc. in the data storage system 250, or store the data, instructions, etc. in the data storage system 250. The data storage system 250 can be placed in the execution device 210, or the data storage system 250 can be an external memory relative to the execution device 210. The calculation module 211 can process the samples obtained by the execution device 210 through the neural network to obtain the prediction result. The specific expression form of the prediction result is related to the function of the neural network.

[0122] It should be noted that the attached Figure 2 This is only an exemplary schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other scenarios, the data storage system 250 may also be placed in the execution device 210.

[0123] The target model / rule 201 trained by the training module 202 can be applied to different systems or devices, such as mobile phones, tablet computers, laptops, augmented reality (AR) / virtual reality (VR), vehicle terminals, etc., and can also be servers or cloud devices.

[0124] The target model / rule 201 in the embodiment of the present application may be a model for executing the action generation method provided in the present application, that is, the target model / rule 201 may be a neural network for action generation provided in the present application. Specifically, the model provided in the embodiment of the present application may include one or more networks such as CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN), etc.

[0125] See attached Figure 3 , the embodiment of the present application also provides a system architecture 300. The execution device 210 is implemented by one or more servers, and optionally cooperates with other computing devices, such as data storage, routers, load balancers, etc.; the execution device 210 can be arranged on a physical site, or distributed on multiple physical sites. The execution device 210 can use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the following of the present application Fig.12 Corresponding steps of a training method for a computing device.

[0126] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0127] The local device of each user can interact with the execution device 210 through a communication network of any communication mechanism / communication standard, and the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes, but is not limited to: a fifth generation mobile communication technology (5th-Generation, 5G) system, a long term evolution (long termevolution, LTE) system, a global system for mobile communication (global system for mobile communication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (near field communication, NFC) in any one or more combinations. The wired network may include a fiber optic communication network or a network composed of a coaxial cable, etc.

[0128] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device, for example, the local device 301 may provide local data or feedback calculation results to the execution device 210. The local device may also be referred to as a computing device.

[0129] It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 301 implements the functions of the execution device 210 and provides services to its own user, or provides services to the user of the local device 302.

[0130] In some scenarios, the sign language digital human can express the meaning of the text through sign language movements, so that users can obtain information through the sign language movements of the sign language digital human.

[0131] In some schemes, the text can be converted into sign language word order through machine translation, and then the final sign language gesture can be obtained through action splicing. For example, gloss (written expression of sign language) can be obtained through rule-based word segmentation, removal of function words and quantifiers, adjustment of word order, etc., and then the corresponding actions are extracted from the motion capture database according to gloss, and the actions are spliced ​​in order. Action splicing mainly considers transition time, that is, the smoothness of action transition. For example, the user can obtain the input corpus "This is a new TV" through voice input or text input, and after sign language transcription, the sign language expression "TV / new / one" is obtained, and then the hand action data of the sign language words "TV", "new" and "one" are extracted respectively, and these action data are spliced ​​to obtain the final sign language gesture action.

[0132] However, rule-based transcription is the method adopted by early translation systems. Most of these rules are obtained through expert experience summary, etc., which has high construction costs and is difficult to fully cover all situations.

[0133] In other solutions, a learning-based approach can be used to directly learn translation rules from paired corpora. For example, a seq2seq network such as transformer can be used to train an end-to-end Chinese text to gloss translation model, and learn this transformation from a large amount of parallel corpora. After obtaining gloss, the corresponding actions are extracted from the motion capture database based on gloss, and the actions are spliced ​​in order.

[0134] However, the actions obtained in the above manner are limited by the action database dictionary, and the interpretation of the meaning expressed by the text may not be accurate enough, so that the final action cannot fully express the specific meaning of the input corpus.

[0135] In other solutions, the text can be directly converted into sign language text gloss through machine translation, and then the gloss can be converted into action through neural network, or the text can be directly converted into action. However, this method may also cause the final action to fail to fully express the specific meaning of the input corpus due to insufficient generalization of the model.

[0136] Therefore, the present application provides an action generation method, which more accurately expresses the meaning represented by the input data and improves the user experience by fusing the vector corresponding to the text and the vector corresponding to the action.

[0137] First, the action generation method provided by the present application can be applied to a variety of scenarios. For example, it can be applied to the user's daily communication locations, such as restaurants, shopping, schools, meetings, travel or social scenes, and the text, voice or image to be expressed can be converted into actions to express it, so that the meaning represented by the text, voice or image can be conveyed to the user through actions in a variety of scenarios.

[0138] For example, the sign language digital person can be used to execute instructions to complete corresponding actions, such as conveying information represented by text, voice or image through the actions of the sign language digital person. In some scenarios, the sign language digital person can be used to convey the meaning of the required expression to the user, so that users who are not convenient to obtain information through voice, text or image can obtain the required information through the actions of the sign language digital person. For example, when the user communicates with the waiter in a restaurant, the waiter can introduce the restaurant's menu to the user through the sign language digital person; when the user goes to a retail store, the mall waiter can use the sign language digital person to inform the user of the specific location of a store, toilet, etc.; for example, users can communicate with each other through the sign language digital person to exchange the information they have learned; for example, when traveling, the sign language digital person can be used to communicate with other users, etc.

[0139] The action generation method provided by this application is introduced in detail below.

[0140] See also Figure 4 , a flowchart of an action generation method provided in this application.

[0141] 401. Obtain input corpus.

[0142] The input corpus may include one or more words.

[0143] The input corpus may be obtained by extracting it locally or from data input by a user. Specifically, a text may be obtained first, and then the text may be used as the input corpus, or the text may be converted or extracted to obtain the input corpus.

[0144] For example, the method of obtaining text may be to extract a piece of text from the local computer and use the text as input corpus; to receive text input by the user and obtain input corpus; or to translate the text input by the user and obtain input corpus after receiving the text input by the user; or to receive an image input by the user and recognize the image and obtain input corpus; or to receive voice data input by the user and recognize the voice data and obtain input text, etc. Therefore, in the embodiments of the present application, input corpus can be obtained in a variety of ways, which can adapt to a variety of scenarios and improve user experience.

[0145] Optionally, after obtaining text from a local computer or through input data, the text can be used directly as input corpus, or the text can be converted or translated and the converted or translated text can be used as input corpus. For example, the word order of words in sign language may be different from the common word order. After obtaining a piece of text, such as "I ride a bicycle", the text can be used directly as input corpus, or the text can be translated into the expression word order of sign language, such as "I / bicycle", which is the input corpus.

[0146] 402. Obtain a word sequence.

[0147] The word sequence may include vectors corresponding to an input corpus, where the input corpus includes one or more words, and each word may correspond to one or more vectors.

[0148] For ease of understanding, the vector corresponding to a word is referred to as a word vector and will not be described in detail below.

[0149] For example, if the word sequence is vectors corresponding to multiple words, the multiple words can form one or more sentences of corpus, that is, input corpus, and each word corresponds to one or more groups of vectors.

[0150] In a possible implementation, the input corpus is converted according to a preset rule to obtain a word sequence, and the input corpus includes text extracted from at least one of text, speech, or image data. For example, a mapping relationship between words and vectors can be preset, and after obtaining the text to be converted into an action, the vector corresponding to each word in the text can be found from the mapping relationship to obtain the word sequence.

[0151] For example, given a text, such as "This is an old mobile phone", according to step 401, corpus that conforms to the sign language word order can be extracted from the text: "mobile phone / old / one", where "mobile phone" is a word, "old" is a word, and "one" is a word with a length of 3. Each word has a corresponding vector representation, such as "mobile phone" corresponds to vector x1, "old" corresponds to vector x2, "one" corresponds to vector x3, and so on.

[0152] In a possible implementation, all or part of the input corpus can also be used as input to a word vector conversion network, and the corresponding word sequence is output. The word vector conversion network is used to convert the input corpus into word vectors. Specifically, the word vector conversion network can include a network for obtaining text in natural language, such as text recognition, image recognition, or machine translation.

[0153] Alternatively, the word vector conversion network may also include a network for natural language processing (NLP). After obtaining the initial vector representation of the input corpus, the initial vector representation is used as the input of the word vector conversion network. The word vector conversion network analyzes each word in the input corpus and its corresponding contextual semantics, thereby obtaining a word vector that can better represent the meaning expressed by the input corpus. For example, each word in the corpus can be converted into an initial vector representation according to a pre-set mapping relationship, each word corresponds to an initial vector representation, and one or more initial representation vectors constitute an input sequence. For example, a section of corpus to be processed, "What's the weather like today", can be obtained. A mapping table can be set in advance, in which each word is set to correspond to a vector, such as "today" corresponds to vector x1, "weather" corresponds to vector x2, etc., and then the input sequence is used as the input of the word vector conversion network to output a word sequence.

[0154] 403. Get action sequence.

[0155] The action sequence may be obtained based on the input corpus or based on the word vector. The action sequence may include a vector corresponding to at least one action, the at least one action corresponds to at least one word in the input corpus, and each word vector may correspond to one or more action vectors (i.e., vectors representing actions).

[0156] Specifically, the action sequence can be determined by a pre-set conversion rule, or the action sequence can be output by a neural network, etc., which can be adjusted according to the actual application scenario.

[0157] In a possible implementation, a rule for converting a word vector or text into an action vector may be preset, such as calculation may be performed according to a preset algorithm, or a mapping relationship between text and action or between a word vector and action may be preset. After obtaining an input corpus or a word sequence, an action vector corresponding to the input corpus or word sequence may be determined according to the rule.

[0158] For example, a mapping relationship between word vectors and action vectors may be pre-set, and after obtaining a word sequence, the action vector corresponding to the input corpus may be searched in the mapping relationship to obtain an action sequence. For another example, a mapping relationship between vocabulary and action vectors may be pre-set, and after obtaining an input corpus, the action vector mapped to the vocabulary in the input corpus may be searched in the mapping relationship to obtain an action sequence.

[0159] In a possible implementation, the action vector corresponding to each word vector in the word sequence or the action vector corresponding to each word in the input corpus can also be output by a neural network to obtain an action sequence. For example, the input corpus or word sequence can be used as the input of the action vector conversion network, and the action vector corresponding to each word vector can be output to obtain the action sequence.

[0160] Optionally, before step 402, the action vector conversion network can be trained using a training set so that the action vector conversion network can convert word vectors into action vectors. The training set may include multiple sample pairs, each of which may include a set of word sequences and action sequences, or a word vector and an action vector, etc. Therefore, in the embodiment of the present application, the word vector can be converted into an action vector by a neural network, and the action vector can be accurately and efficiently output compared to the query mapping relationship.

[0161] It should be noted that if the action sequence is obtained by inputting corpus, step 402 may be executed first, or step 403 may be executed first, or both step 402 and step 403 may be executed simultaneously. The specific steps may be adjusted according to the actual application scenario, and this application does not limit this.

[0162] 404. Fuse the word sequence and the action sequence to obtain a fused sequence.

[0163] After obtaining the word sequence and the action sequence, the word sequence and the action sequence are fused to obtain a fused sequence.

[0164] Specific fusion methods may include splicing, weighted fusion, or splicing after processing, etc. The matching fusion method can be selected according to the actual application scenario, and this application does not limit this.

[0165] For example, the word vectors included in the word sequence and the action vectors included in the action sequence may be concatenated, so that the obtained fused sequence includes both the information included in the word sequence and the information included in the action sequence.

[0166] In a possible implementation, an expression sequence may also be obtained, and the expression sequence may include one or more groups of expression vectors, each group of expression vectors may represent one or more expression actions, and is used to more accurately and vividly express the information represented by the input data. Subsequently, the word sequence, the action sequence, and the expression sequence may be fused to obtain a fused sequence. Thus, the fused sequence may include more information such as words, actions, and expressions, and further, the action generation network may extract more information from the fused sequence, thereby improving the output accuracy of the action generation network.

[0167] 405. The fused sequence is used as the input of the action generation network, and a parameter set is output.

[0168] Among them, the action generation network can be used to output action parameters corresponding to the input vector, that is, the parameter set may include parameters representing the action, such as parameters representing the direction of movement, the position of the joints or the distance of movement. The action parameters can be used to render one or more frames of action images, thereby converting the word sequence into action, so that the user can obtain the required information through the action image.

[0169] Therefore, in the implementation mode of the present application, after obtaining the action sequence, the action sequence and the word sequence can be fused, so that the obtained fused sequence can include the information included in the action sequence and the word sequence, so that the neural network can output more accurate and coherent parameters corresponding to the action based on the information included in the action sequence and the word sequence, so that the final action can accurately express the information expressed by the input data, thereby improving the user experience.

[0170] In a possible implementation, the parameter set may also include expression parameters for rendering expressions, so that the user can obtain context information that is more relevant to the input sequence, making the information obtained by the user more accurate and improving the user experience.

[0171] In a possible implementation, before step 404, the action generation network may be trained using a training set to obtain a trained action generation network, so that the action generation network can output accurate action parameters.

[0172] Optionally, the training set may include multiple sample pairs, each sample pair may include a set of word sequences, and action parameters corresponding to the word sequences, such as each sample pair may include a word vector of a complete corpus, and an action vector of a complete set of actions corresponding to the corpus, so that the actions corresponding to the action parameters output by the trained action generation network can be made more coherent. For example, compared to each sample pair including only one word vector and one or more corresponding action vectors, the sample pairs provided in this embodiment can enable the action generation network to learn more coherent actions, so that the actions corresponding to the parameters output by the action generation network are also more coherent, thereby improving the user experience.

[0173] In addition, the sample pairs in the training set can also include word vectors and corresponding action vectors, so that the action generation network can learn the action corresponding to each word and improve the output accuracy of the action generation network. The specific details can be adjusted according to the actual application scenario.

[0174] In a possible implementation, if other vectors are also fused into the fusion vector, such as expression vectors or other vectors that can represent the information included in the input data, the action generation network can also output expression actions or other parameters corresponding to actions that can express word sequences and action sequences. Accordingly, when training the action generation network, the sample pairs in the training set used can also include expressions or other expressions that can express word sequences and actions, so that the action generation network outputs more types of parameters, and uses more actions to express the information represented by the input data or the contextual semantics of each word, so that users can get more and more accurate information through the actions rendered by the parameter set, thereby improving the user experience.

[0175] The above describes the process of the action generation method provided by this application. For ease of understanding, the following is a description of the process of the action generation method provided by this application. Figure 4 The method provided in this application is introduced in more detail based on the method flow and specific application scenarios.

[0176] It should be noted that the present application exemplarily introduces the conversion of text into sign language actions as an example, and the text can also be converted into other actions. The specific action type can be adjusted according to the actual application scenario, and the present application does not limit this.

[0177] See also Figure 5 , a flowchart of another action generation method provided by the present application.

[0178] 501. Text acquisition.

[0179] Among them, the text can be obtained through various ways, such as voice recognition, text input, image recognition or machine translation to obtain the input corpus to be converted.

[0180] For example, the action generation method provided in the present application can be deployed in the user's terminal in the form of an application program (APP). The user can use the terminal to take pictures, and the terminal can recognize the captured image and identify the text included in the image. Alternatively, the user can use the microphone of the terminal to input voice, and the terminal obtains the text input by the user through voice recognition. For another example, the user can directly input the corpus in the APP interface. For another example, the user can input the text in the first language in the APP interface, and the terminal recognizes the text in the first language to obtain the text in the second language, etc.

[0181] 502. Sign language interpretation.

[0182] After obtaining the input corpus, the input corpus can be translated into a text that can be expressed in sign language, that is, a sign language text, which includes one or more sign language words.

[0183] Generally, sign language is an independent form of expression, with independent word order, language structure or grammatical rules. For example, there are usually no quantifiers in sign language, and numerals are placed after nouns. The text "There are five people in my family" is expressed as "I / family / people / five" in sign language. Another example is the verbalization of nouns. The Chinese text "I ride a bicycle" is expressed as "I / bicycle" in sign language.

[0184] In the sign language expression, some words have specific attributes, such as special expressions. Information such as body orientation can also be marked in the sign language expression. For example, the Chinese text "Why didn't you buy a down jacket?", in the sign language expression, with a question expression "Down jacket / buy / no / why (question)".

[0185] In the implementation of the present application, these sign language rules can be collected by experts, and the sign language translation rules can be directly constructed in a rule-based manner; or a large number of parallel corpora of paired Chinese text-to-sign language written expressions can be collected, and a sign language translation model can be trained in a deep learning-based manner to extract translation rules, and the Chinese text can be directly converted into a sign language text. For example, when parallel corpora are unavailable or the corpora are small, the performance of the sign language translation module constructed in a rule-based manner can be better; when there are a large number of parallel corpora, a sign language translation module constructed in a deep learning-based manner can be used to obtain a sign language translation network with better performance.

[0186] 503. Action conversion.

[0187] After the sign language text is obtained through sign language translation, the action corresponding to the sign language text can be obtained to obtain the sign language action.

[0188] Specifically, the sign language action can be obtained by searching the action corresponding to the sign language vocabulary in the sign language action database. For example, the action database can be pre-set, and some sign language words and their corresponding sign language actions in a common sign language dictionary can be saved in the action database. After the sign language vocabulary is obtained, the sign language action corresponding to the sign language vocabulary can be searched in the action database.

[0189] 504. Multimodal fusion.

[0190] After obtaining the sign language vocabulary and sign language movements, the sign language vocabulary can be converted into word vectors, and the sign language movements can be converted into movement vectors. Then, the word vectors and movement vectors are fused to obtain a fused vector, so that the obtained fused vector can include both the information contained in the sign language vocabulary and the information expressed by the sign language movements.

[0191] The specific method of fusing the word vector and the action vector can be concatenation, concatenation after processing the word vector and the action vector, or weighted fusion, which can be adjusted according to the actual application scenario. For example, the word vector can be represented as x1, and the action vector can be represented as y1. By concatenating x1 and y1, the fused vector x1y1 can be obtained.

[0192] Specifically, the specific method of converting sign language vocabulary and sign language movements into vectors can be to convert them according to a pre-set mapping relationship, or to convert them through a neural network.

[0193] For example, the mapping relationship between sign language vocabulary and word vectors can be pre-set, such as "today" corresponds to vector x1, "weather" corresponds to vector x2, etc. After obtaining the sign language vocabulary, the word vector corresponding to the sign language vocabulary can be searched according to the pre-set mapping relationship to obtain the word sequence.

[0194] For another example, a word vector conversion network may be used to output the word vector. The word vector conversion network may include a fully connected network or a multi-layer perceptron network to convert the sign language vocabulary expressed in text into a word vector.

[0195] For example, a motion vector conversion network can be used to output a motion vector. The motion vector conversion network can also be implemented by a fully connected network or a multi-layer perceptron network to convert sign language movements into motion vectors.

[0196] For example, Figure 6 As shown, after obtaining the sign language vocabulary and sign language movements, the sign language vocabulary is used as the input of the word vector conversion network to output the word vector, and the sign language movement is used as the input of the action vector conversion network to output the action vector. Then, the word vector and the action vector are fused through the fusion network to obtain the fused vector.

[0197] In addition, usually a sign language word may correspond to multiple frames of sign language actions, so the number of frames of the word vector needs to be aligned with the number of frames of the action vector before the word vector and the action vector are fused. The specific fusion method may include: aligning each sign language word with one or more corresponding action vectors, or, when a sign language word corresponds to multiple action vectors, the multiple vectors can be fused into a fused action vector, and then the word vector and the fused action vector are aligned.

[0198] For example, Figure 7As shown, the first sign language vocabulary corresponds to 3 frames of action, the second sign language vocabulary has no corresponding action found in the action database, and the third sign language vocabulary has 2 frames of action. The word vectors g1, g2, and g3 of these three sign language vocabulary can be obtained respectively; through the action vector conversion network, the action vectors m1f1, m1f2, and m1f3 corresponding to the first sign language vocabulary and the action vectors m3f1 and m3f2 corresponding to the third sign language vocabulary can be obtained. The second sign language vocabulary has no corresponding action retrieved in the action database, so a placeholder vector is used to represent its action vector. In a possible implementation, the word vector is directly spliced ​​with the action vector of each frame, and the splicing result is input into the fusion network to obtain the fusion vector. The number of frames of the fusion vector is consistent with the number of frames of the action vector input into the fusion network. The result of the fusion of the word vector g1 and the action vector m1f1 is G1f1, the result of the fusion of the word vector g1 and the action vector m1f2 is G1f2, the result of the fusion of the word vector g2 and the placeholder vector is G2f1, and so on. Then the word vector and the action vector can be concatenated, and the concatenated result can be input into a fully connected layer and a ReLU layer for processing to obtain the final fused vector.

[0199] For example, Figure 8 As shown in , multiple action vectors corresponding to a sign language vocabulary can be merged into one action vector. For example, the first sign language vocabulary corresponds to 3 frames of action, the second sign language vocabulary has no corresponding action in the action database, and the third sign language vocabulary corresponds to 2 frames of action. The word vectors g1, g2, and g3 of these three sign language vocabulary can be obtained respectively. Through the action vector conversion network, multiple action vectors corresponding to the first sign language vocabulary can be obtained and fused to obtain m1, and multiple action vectors corresponding to the third sign language vocabulary can be fused to obtain m3. Then, g1 and m1 are fused to obtain G1, g2 and m2 are fused to obtain G2, and g3 and m3 are fused to obtain G3.

[0200] Furthermore, in some possible implementations, after obtaining the input corpus, in addition to sign language vocabulary and sign language movements, other information such as facial expressions, body orientation, and the like may also be obtained from the input corpus.

[0201] Specifically, the type of expression can be obtained based on the input corpus, and the corresponding expression vector can be generated to obtain an expression sequence. The expression sequence can include one or more expression corresponding vectors. The word sequence, action sequence and expression sequence can be fused to obtain a fused sequence, so that the fused sequence can include not only the information carried by the word sequence and the action sequence, but also the expression-related information carried by the expression sequence.

[0202] Specific methods for obtaining the type of expression based on the input corpus may include generating matching expressions based on the contextual semantics of the input corpus, determining matching expressions based on information carried in the input corpus, such as modal particles, punctuation marks, etc., or determining matching expressions based on the contextual semantics of the input corpus and the information carried. For example, if the input corpus carries a "?", and combined with the contextual semantics of the input corpus, it is determined that the current expression should be a puzzled expression. The vector representing each expression or the generation rule of the vector can be pre-set, and after determining one or more expressions that match the input corpus, the corresponding expression vector can be obtained.

[0203] For example, Fig. 9 As shown, the word vector of the first sign language vocabulary is g1, the first frame action vector of the first sign language vocabulary corresponding to the action is m1f1, and the first frame expression vector corresponding to the first sign language vocabulary is e1f1. Taking the first frame as an example, the word vector g1, the result of the fusion of the action vector m1f1 and the expression vector e1f1 is G1f1, and so on.

[0204] 505. Sign language generation.

[0205] After obtaining the fusion vector, the fusion vector can be used as the input of the action generation network, and a parameter set can be output, such as parameters representing the arm movement direction, hand position, joint morphology, etc. of the sign language digital human. After generating the action parameters of the sign language, the action parameters can be used for rendering, and the rendered image corresponding to one or more frames of the sign language action can be obtained and displayed in the display interface.

[0206] For example, the sign language action sequence corresponding to the input vector can be output through the action generation network, which may include one or more information of the sign language identification, movement direction, movement amplitude, etc. If each sign language action sequence saves the corresponding sign language action, the rendered data can be directly extracted for playback, or it can be rendered and played in real time based on the sign language action sequence. The specific method can be adjusted according to the actual application scenario, and this application does not limit the rendering method.

[0207] In addition, if the fusion vector also incorporates an expression vector, the output of the action generation network may also include expression parameters, which can be used for rendering to obtain an image that includes an expression.

[0208] Specifically, after obtaining the fused vector, the fused vector can be used as the input of a sequence-to-sequence transformation (seq2seq) network, i.e., an action generation network, and outputs a parameter set. The seq2seq network can generate a parameter set in an autoregressive or non-autoregressive manner.

[0209] In a possible implementation, the parameter set may be outputted in an autoregressive manner, such as using a deep self-attention transformer network as an encoder of the fusion vector and a generator of the parameter set.

[0210] For example, Fig.10 As shown, where x n represents the nth fusion vector in the fusion sequence, N is the total length of the fusion vector, y t The gesture parameters and expression parameters of the sign language (i.e., the parameters included in the parameter set). The Transformer encoder (encoding component) can use the self-attention mechanism to read the information carried by the word vector in the fusion vector. For example, the Transformer encoder can include multiple self-attention modules, and the SA module can be used to calculate the association degree based on the input vector, such as the degree of association between each word in the corpus and one or more adjacent words, and then fuse the input vector and the association degree to obtain the output result of the SA module.

[0211] Then the read information is input to the Transformer Decoder (decoding component). u and position code c u For example, the uth frame can be any frame of sign language action, and the Transformer Decoder can generate the action of the u+1th frame. and position encoding When the position code is generated When the preset threshold is reached, the synthesis of the sign language action of the next frame is stopped. Usually, the output of the Transformer Decoder is a real vector, and the floating point number can be transformed into a usable parameter through a linear transformation. The linear transformation layer can include a fully connected layer, which is used to project the vector generated by the Transformer Decoder into a vector of logits to obtain the action parameters.

[0212] In another possible implementation, the parameter set may be generated in a non-autoregressive manner. Compared with the autoregressive manner, the difference is that in the process of generating the parameter set in the non-autoregressive manner, the total length of the sign language action may be estimated in advance, and then the position encoding of each sign language action may be determined based on the length.

[0213] For example, Fig.11 As shown, for the above Fig.10The similar parts are not introduced here. The difference is that T comes from the length estimation module, or is obtained according to the input data given by the user. The length estimation module can be a small neural network used to estimate the length of the input data. The input of the module can be the output of the Transformer Encoder, and the output of the module is the length of the sign language action corresponding to the input corpus. When the length of the generated sign language action is determined, the sign language action can be directly estimated by a non-autoregressive method, thereby reducing the inference time.

[0214] Therefore, in the embodiment of the present application, after obtaining the word vector and the action vector, the word vector and the action vector can be fused to obtain a fused vector. The fused vector can include the information contained in the word vector and the action vector, so that the fused vector containing more information can be used as the input of the seq2seq network, so that more accurate sign language action output can be obtained by combining the contextual semantic information of the word vector.

[0215] The action generation method provided in this application can be implemented through a neural network.

[0216] Exemplarily, the present application provides a neural network for action generation, such as Fig.12 As shown, the neural network may specifically include:

[0217] The text acquisition module 1201 is used to acquire input corpus, which includes at least one word.

[0218] A word vector conversion network 1202, used to obtain a word sequence according to an input corpus, wherein the word sequence includes a vector corresponding to at least one word;

[0219] An action vector conversion network 1203, used to obtain an action sequence, wherein the action sequence includes a vector corresponding to at least one action, and the at least one action includes an action corresponding to at least one word;

[0220] A fusion network 1204 is used to fuse the word sequence and the action sequence to obtain a fusion sequence;

[0221] The action generation network 1205 uses the fusion sequence as input of the action generation network and outputs a parameter set, wherein the parameter set includes action parameters. The action parameters are used to render an action image. The action generation network is used to convert the input vector into parameters related to the action.

[0222] In a possible implementation, the parameter set further includes expression parameters, and the expression parameters are used to render expression actions corresponding to at least one action.

[0223] In a possible implementation, the above model may further include:

[0224] An expression vector conversion network 1206 is used to obtain an expression sequence according to the input corpus, wherein the expression sequence includes a vector corresponding to at least one expression;

[0225] The fusion network 1204 is specifically used to fuse the word sequence, the action sequence and the expression sequence to obtain a fusion sequence.

[0226] In a possible implementation, the text acquisition module 1201 is specifically used to extract text from at least one of text, speech or image data to obtain input corpus. For example, the text acquisition module may include a text recognition network, a speech recognition network or an image recognition network.

[0227] In a possible implementation, a training set may be used to train the neural network, where the training set includes a plurality of sample pairs, each sample pair including a set of word sequences and at least one corresponding set of action parameters.

[0228] In addition, the present application also provides a training method, such as Fig.13 As shown, the training method may include:

[0229] 1301. Use the training set to train the neural network to obtain a trained neural network.

[0230] The training set includes multiple sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters, the neural network is used to fuse the input action sequence and word sequence to obtain a fused sequence, and output a parameter set according to the fused sequence, the word sequence includes a vector corresponding to at least one word in the input corpus, the action sequence includes a vector corresponding to at least one action, at least one action includes an action corresponding to at least one word, the parameter set includes action parameters, and the action parameters are used to render an action image.

[0231] Therefore, in the implementation manner of the present application, when training the neural network, each sample pair included in the training set used may include a set of word sequences and corresponding complete action parameters of the action, so that the actions output by the neural network are more coherent, thereby improving the user experience.

[0232] In one possible implementation, a neural network is specifically used to fuse word sequences, action sequences, and expression sequences to obtain a fused sequence, wherein the expression sequence is obtained based on an input corpus, and the expression sequence includes a vector corresponding to at least one expression; the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0233] Therefore, in the implementation manner of the present application, the neural network can also output expression parameters, so that an expression image can be rendered, so that the user can more accurately know the specific information of the input corpus through the expression image, thereby improving the user experience.

[0234] In a possible implementation, before using the training set to train the neural network, the neural network is also pre-trained using a data set to obtain a pre-trained neural network, and the data set may include multiple sample pairs, each sample pair may include a vocabulary or word vector and a corresponding action parameter, and then the training set may be used to fine-tune the pre-trained neural network to obtain a trained neural network.

[0235] Therefore, in the implementation mode of the present application, in the pre-training stage, word vectors and their corresponding action parameters can be used for training, so as to ensure that the final neural network can output accurate actions. In the fine-tuning stage, word sequences and their corresponding complete action parameters can be used for training, so that the final neural network can output more coherent action parameters on the basis of accurate output, thereby improving the user experience.

[0236] Of course, in both the pre-training stage and the fine-tuning stage, the aforementioned training set can be used to train the neural network, so that the output of the final neural network is accurate and the movements are more coherent, thereby improving the user experience. Specifically, the samples used in each stage can be selected according to the actual application scenario, and this application does not limit this.

[0237] Generally, a variety of samples can be used as training sets for training neural networks. The following takes the application in the scenario of converting text into sign language as an example to illustrate several possible training sets.

[0238] Training set 1 (the aforementioned data set)

[0239] The first training set may include an action database consisting of sign language actions and expression data corresponding to commonly used sign language vocabulary.

[0240] Among them, the commonly used sign language vocabulary can come from some general sign language dictionaries or statistics of users' daily conversations, and the sign language actions corresponding to the sign language vocabulary in the action database can be marked with the start frame and the end frame.

[0241] It is also possible to increase the commonly used sign language vocabulary in the sign language database by making sentences, and construct a large number of training samples by fusion of actions. Fig.14As shown, a certain vocabulary matches two actions, namely Action 1 and Action 2. Each action has a starting action and an ending action, such as raising an arm or lowering an arm, etc. An action transition can be generated between Action 1 and Action 2. The ending action of Action 1 and the starting action of Action 2 can be combined. For example, if the ending action is lowering the arm and the starting action is raising the arm, they can be combined into an action of placing the arm in the middle position or other actions that smooth Action 1 and Action 2, so that the actions output by the finally trained model are smoother.

[0242] Therefore, by collecting information such as some common sign language vocabularies and their corresponding actions or expressions, training data can be collected at low cost, improving the performance of the model. And, by adding a training set of transition actions for model training, the actions output by the model are also more coherent, improving the user experience.

[0243] Training set two (i.e., the aforementioned training set)

[0244] The difference from the aforementioned training set one is that the sample pairs in training set two can include a piece of corpus and the parameters of the complete sign language actions corresponding to the corpus. For example, in a certain sample pair, it can include the sign language vocabulary "I / family / people / five", and the parameters of a complete set of sign language actions corresponding to this piece of corpus.

[0245] Therefore, when using training set two for model training, the model can learn the corpus and its corresponding complete sign language actions, so that the model can output the parameters of more coherent sign language actions.

[0246] In addition, it is also possible to combine the aforementioned training set one and training set two for model training. Generally, since the acquisition cost of training set one is relatively low, a large amount of training data can be obtained quickly; training set two can obtain the true value data of the complete corpus. Therefore, this application can use a two-stage training method for model training. Whether the data volume of the first data set increases or the data volume of the second data set increases, it is ultimately beneficial to optimize the final performance.

[0247] For example, in the pre-training stage, the model is pre-trained using the data constructed from training set one, so as to obtain an accurate output model; in the fine-tuning stage, training set two is used for fine-tuning, and on the basis of accurate output, a model with more coherent sign language actions output is obtained. Of course, in both the pre-training stage and the fine-tuning stage, training set two can be used for training, so that the sign language actions output by the final neural network are more coherent, improving the user experience.

[0248] The above has introduced in detail the action generation method, the training method process, and the neural network provided by this application. For ease of understanding, the following will give an exemplary introduction to some specific application scenarios and the achieved effects of this application.

[0249] For example, the accuracy of sign language generation results can be measured by the dynamic time warping (DTW) distance. The DTW algorithm can align one high-dimensional time series to another high-dimensional time series. If two high-dimensional time series are closer, the DTW distance between the two high-dimensional time series will be smaller.

[0250] In one scenario, a motion database of 283 sign language words is collected. Then, this motion database is used to construct the training set Random10k and the test set Random1k. The training set Random10k is constructed as follows: 10,000 sentences are obtained by randomly constructing sentences, and the longest sentence is 6 words. Then, the sign language GT of the entire sentence is obtained by using the motion splicing and fusion method. During training, 30% of the words in the motion database have no corresponding actions.

[0251] The test set Random1k is constructed as follows: 1,000 sentences are obtained by randomly constructing sentences, with the longest sentence length being 6 words, and then the sign language movements of the entire sentence are obtained by using action splicing and fusion.

[0252] Actions can be extracted from 100 real sign sentences to construct the test set GT100.

[0253] A variety of solutions were compared, and solution one included: generating corresponding sign language actions based only on text information; solution two included: generating corresponding sign language actions based on action information. The solution provided in this application inputs text information and action information.

[0254] The effects achieved respectively can be shown in Table 1. Therefore, the solution provided by the present application is superior to the pure text solution and the pure action solution. Moreover, even when the text pair does not include sign language actions, the output effect of the model can be better.

[0255] Validation set Solution 1 Solution 2 This application plan DTW Plain text Pure Action Text + Action Random1k 2.33 6.32 1.11 GT100 14.55 13.15 9.15

[0256] Table 1

[0257] In another scenario, a motion database of 283 sign language words was collected. This motion database was then used to construct the training set Random10k and the test set Random1k. The training set Random10k was constructed as follows: 10,000 sentences were obtained by randomly constructing sentences, and the longest sentence was 6 words. Then, the motion splicing and fusion method was used to obtain the sign language motion of the entire sentence. During training, it was assumed that all words in the motion database had corresponding motions.

[0258] The test set Random1k is constructed as follows: 1,000 sentences are obtained by randomly constructing sentences, with the longest sentence length being 6 words, and then the sign language movements of the entire sentence are obtained by using action splicing and fusion.

[0259] Actions can be extracted from 100 real sign sentences to construct the test set GT100.

[0260] A variety of solutions were compared, and solution one included: generating corresponding sign language actions based only on text information; solution two included: generating corresponding sign language actions based on action information. The solution provided in this application inputs text information and action information.

[0261] The effects achieved respectively can be shown in Table 2. Experiments show that the output effect of the solution provided by this application is better than the output effect of the pure text solution and the pure action solution. When the action corresponding to the sign language text is obtained, the action information is input into the model, so that the model can focus on learning better action connection.

[0262] Validation set Solution 1 Solution 2 This patent solution DTW Plain text Pure Action Text + Action Random1k 2.376 0.860 0.78 GT100 14.55 9.35 8.46

[0263] Table 2

[0264] The above describes the method and model provided by the present application. The following is a detailed introduction to the device provided by the present application.

[0265] See also Fig.15 , a schematic diagram of the structure of an action generation device provided in the present application, the action generation device is used to execute the method steps of the first aspect or any embodiment of the first aspect, the action generation device may include:

[0266] A text acquisition module 1501 is used to acquire input corpus, where the input corpus includes at least one word;

[0267] A word vector conversion module 1502, configured to obtain a word sequence according to an input corpus, wherein the word sequence includes a vector corresponding to at least one word;

[0268] An action vector conversion module 1503, used to obtain an action sequence, wherein the action sequence includes a vector corresponding to at least one action, and the at least one action includes an action corresponding to at least one word;

[0269] A fusion module 1504 is used to fuse the word sequence and the action sequence to obtain a fused sequence;

[0270] The output module 1505 is used to use the fusion sequence as the input of the action generation network, and output a parameter set, wherein the parameter set includes action parameters, and the action parameters are used to render the action image. The action generation network is used to convert the input vector into parameters related to the action.

[0271] In a possible implementation, the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0272] In a possible implementation, the device further includes:

[0273] An expression vector conversion module 1506 is used to obtain an expression sequence according to the input corpus, wherein the expression sequence includes a vector corresponding to at least one expression;

[0274] The fusion module is specifically used to fuse word sequences, action sequences and expression sequences to obtain a fused sequence.

[0275] In a possible implementation, the device further includes:

[0276] The training module 1507 is used to train the action generation network using a training set before using the fused sequence as the input of the action generation network and outputting a parameter set. The training set includes multiple sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters.

[0277] In a possible implementation, the word vector conversion module 1502 is specifically used to:

[0278] Convert the input corpus or word sequence according to preset rules to obtain a word sequence;

[0279] Alternatively, the input corpus or word sequence is used as the input of the word vector conversion network, and the word vector conversion network is used to convert the corpus into the corresponding vector.

[0280] In a possible implementation, the action vector conversion module 1503 is specifically used to query the action corresponding to the input corpus or word sequence from a preset action database to obtain an action sequence; or, use the input corpus or word sequence as the input of the action vector conversion network to output the action sequence.

[0281] In a possible implementation, the text acquisition module 1501 is specifically configured to extract text from at least one of text, speech or image data, and use the text as input corpus or convert the text to obtain input corpus.

[0282] See also Fig.16 , a structural schematic diagram of another action generation device provided by the present application is described as follows.

[0283] The action generation device may include a processor 1601 and a memory 1602. The processor 1601 and the memory 1602 are interconnected via a line. The memory 1602 stores program instructions and data.

[0284] The memory 1602 stores the aforementioned Figure 4-Figure 11 The program instructions and data corresponding to the steps in .

[0285] Processor 1601 is used to execute the above Figure 4-Figure 11 The method steps are performed by the action generation device shown in any embodiment.

[0286] Optionally, the action generating device may further include a transceiver 1603 for receiving or sending data.

[0287] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 4-Figure 11 The illustrated embodiments describe steps in a method.

[0288] Optionally, the aforementioned Fig.16 The action generating device shown in is a chip.

[0289] The embodiment of the present application also provides an action generation device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 4-Figure 11 The method steps are performed by the action generation device shown in any embodiment.

[0290] See also Fig.17 , a schematic diagram of the structure of a training device provided in the present application, the training device is used to perform the method steps in the first aspect or any embodiment of the first aspect, the training device may include:

[0291] The training module 1701 is used to train the neural network using the training set to obtain the trained neural network. The training set includes multiple sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters. The neural network is used to fuse the input action sequence and word sequence to obtain a fused sequence, and output a parameter set according to the fused sequence. The word sequence includes a vector corresponding to at least one word in the input corpus, the action sequence includes a vector corresponding to at least one action, at least one action includes an action corresponding to at least one word, and the parameter set includes action parameters, which are used to render an action image.

[0292] In one possible implementation, a neural network is specifically used to fuse word sequences, action sequences, and expression sequences to obtain a fused sequence, wherein the expression sequence is obtained based on an input corpus, and the expression sequence includes a vector corresponding to at least one expression; the parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

[0293] In a possible implementation, the training module 1701 is further used to pre-train the neural network using a data set before training the neural network using the training set to obtain a pre-trained neural network. The data set may include multiple sample pairs, each sample pair may include a vocabulary or word vector and a corresponding action parameter. Subsequently, the training set may be used to fine-tune the pre-trained neural network to obtain a trained neural network.

[0294] It can be understood that the neural network can be used to perform the aforementioned Figure 4-Figure 11 Corresponding method steps, or the neural network can be the aforementioned Fig.12 The neural network used for action generation in .

[0295] See also Fig.18 , a structural schematic diagram of another training device provided in the present application is described as follows.

[0296] The training device may include a processor 1801 and a memory 1802. The processor 1801 and the memory 1802 are interconnected via a line. The memory 1802 stores program instructions and data.

[0297] The memory 1802 stores the aforementioned Fig.16 The program instructions and data for the steps performed by the training device.

[0298] Processor 1801 is used to execute the above Fig.16 Method steps performed by the training device.

[0299] Optionally, the training device may further include a transceiver 1803 for receiving or sending data.

[0300] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 4-Figure 11 The illustrated embodiments describe steps in a method.

[0301] Optionally, the aforementioned Fig.18 The training device shown in is a chip.

[0302] The embodiment of the present application also provides a training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 4-Figure 11 The method steps are performed by the training device shown in any embodiment.

[0303] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the functions of the above-mentioned processor 1601, processor 1801, or processor 1601, processor 1801. When a memory is integrated in the digital processing chip, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the action generation device in the above-mentioned embodiment according to the program code stored in the external memory.

[0304] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figure 4-Figure 11 or Fig.13 The illustrated embodiment describes the steps performed by the action generating device in the method.

[0305] The action generation device or training device provided in the embodiment of the present application can be a chip, and the chip includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figure 4-Figure 11 or Fig.13 The action generation method described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0306] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0307] For example, see Fig.19 , Fig.19 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 190, NPU 190 is mounted on the host CPU (Host CPU) as a coprocessor, and the Host CPU assigns tasks. The core part of the NPU is the operation circuit 1903, which is controlled by the controller 1904 to extract matrix data from the memory and perform multiplication operations.

[0308] In some implementations, the operation circuit 1903 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 1903 is a two-dimensional systolic array. The operation circuit 1903 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1903 is a general-purpose matrix processor.

[0309] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1902 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1901 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 1908.

[0310] The unified memory 1906 is used to store input data and output data. The weight data is directly transferred to the weight memory 1902 through the direct memory access controller (DMAC) 1905. The input data is also transferred to the unified memory 1906 through the DMAC.

[0311] The bus interface unit (BIU) 1910 is used for the interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 1909 .

[0312] The bus interface unit 1910 (BIU) is used for the instruction fetch memory 1909 to obtain instructions from the external memory, and is also used for the storage unit access controller 1905 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0313] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1906 or to transfer weight data to the weight memory 1902 or to transfer input data to the input memory 1901.

[0314] The vector calculation unit 1907 includes multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0315] In some implementations, the vector calculation unit 1907 can store the processed output vector to the unified memory 1906. For example, the vector calculation unit 1907 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1903, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1907 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1903, for example, for use in a subsequent layer in a neural network.

[0316] An instruction fetch buffer 1909 connected to the controller 1904 is used to store instructions used by the controller 1904;

[0317] Unified memory 1906, input memory 1901, weight memory 1902 and instruction fetch memory 1909 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0318] Among them, the operations of each layer in the recurrent neural network can be performed by the operation circuit 1903 or the vector calculation unit 1907.

[0319] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the above Figure 4-Figure 11 or Fig.13 Method of procedure.

[0320] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0321] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0322] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0323] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0324] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0325] Finally, it should be noted that the above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An action generation method, characterized in that: include: Acquire input corpus, where the input corpus includes at least one word; Acquire a word sequence according to the input corpus, wherein the word sequence includes a vector corresponding to the at least one word; Acquire an action sequence, wherein the action sequence includes a vector corresponding to at least one action, wherein the at least one action includes an action corresponding to the at least one word, and the at least one action is a sign language action; fusing the word sequence and the action sequence to obtain a fused sequence, wherein the fused sequence is obtained by aligning the word sequence and the action sequence and then fusing them; The fused sequence is used as the input of an action generation network, and a parameter set is output. The parameter set includes action parameters, and the action parameters are used to render an action image. The action generation network is used to convert the input vector into parameters related to the action. The action image includes an image of a sign language action, and the sign language action is used to express the meaning of the input corpus.

2. The method according to claim 1, characterized in that The parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

3. The method according to claim 2, characterized in that The method further comprises: Acquire an expression sequence according to the input corpus, wherein the expression sequence includes a vector corresponding to at least one expression; The fusing the word sequence and the action sequence to obtain a fused sequence includes: The word sequence, the action sequence and the expression sequence are fused to obtain the fused sequence.

4. The method according to any one of claims 1 to 3, characterized in that Before using the fused sequence as an input of the action generation network and outputting a parameter set, the method further includes: The action generation network is trained using a training set, where the training set includes a plurality of sample pairs, each sample pair including a set of word sequences and at least one corresponding set of action parameters.

5. The method according to any one of claims 1 to 3, characterized in that: The acquiring a word sequence according to the input corpus comprises: Converting the input corpus according to a preset rule to obtain the word sequence; Alternatively, the input corpus is used as input of a word vector conversion network, and the word sequence is output. The word vector conversion network is used to convert the corpus into corresponding vectors.

6. The method according to any one of claims 1 to 3, characterized in that The acquisition action sequence includes: Querying an action corresponding to the input corpus or the word sequence from a preset action database to obtain the action sequence; Alternatively, the input corpus or the word sequence is used as the input of an action vector conversion network, and the action sequence is output.

7. The method according to any one of claims 1 to 3, characterized in that The obtaining of input corpus includes: Extracting text from at least one of text, speech or image data; The text is used as the input corpus, or the text is converted to obtain the input corpus.

8. An action generating device, characterized in that: include: A text acquisition module, used to acquire input corpus, wherein the input corpus includes at least one word; A word vector conversion module, used for obtaining a word sequence according to the input corpus, wherein the word sequence includes a vector corresponding to the at least one word; an action vector conversion module, used to obtain an action sequence, wherein the action sequence includes a vector corresponding to at least one action, wherein the at least one action includes an action corresponding to the at least one word, and the at least one action is a sign language action; A fusion module, used for fusing the word sequence and the action sequence to obtain a fused sequence, wherein the fused sequence is obtained by aligning the word sequence and the action sequence and then fusing them; An output module is used to use the fusion sequence as an input of an action generation network and output a parameter set, wherein the parameter set includes action parameters, and the action parameters are used to render an action image. The action generation network is used to convert the input vector into parameters related to the action, and the action image includes an image of a sign language action, and the sign language action is used to express the meaning of the input corpus.

9. The device according to claim 8, characterized in that The parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

10. The device according to claim 9, characterized in that The device also includes: An expression vector conversion module, used for obtaining an expression sequence according to the input corpus, wherein the expression sequence includes a vector corresponding to at least one expression; The fusion module is specifically used to fuse the word sequence, the action sequence and the expression sequence to obtain the fusion sequence.

11. The device according to any one of claims 8 to 10, characterized in that The device also includes: A training module is used to train the action generation network using a training set before using the fusion sequence as the input of the action generation network and outputting a parameter set, wherein the training set includes a plurality of sample pairs, each sample pair includes a set of word sequences and at least one corresponding set of action parameters.

12. The device according to any one of claims 8 to 10, characterized in that The word vector conversion module is specifically used for: Converting the input corpus according to a preset rule to obtain the word sequence; Alternatively, the input corpus is used as input of a word vector conversion network, and the word sequence is output. The word vector conversion network is used to convert the corpus into corresponding vectors.

13. The device according to any one of claims 8 to 10, characterized in that The action vector conversion module is specifically used to query the action corresponding to the input corpus or the word sequence from a preset action database to obtain the action sequence; or, use the input corpus or the word sequence as the input of the action vector conversion network to output the action sequence.

14. The device according to any one of claims 8 to 10, characterized in that The text acquisition module is specifically used to extract text from at least one of text, voice or image data; use the text as the input corpus, or convert the text to obtain the input corpus.

15. A training method, characterized in that: include: A neural network is trained using a training set to obtain a trained neural network, wherein the training set includes a plurality of sample pairs, each sample pair includes a set of word sequences and corresponding at least one set of action parameters, the neural network is used to fuse an input action sequence and a word sequence to obtain a fused sequence, the fused sequence is obtained by aligning the word sequence and the action sequence and then fusing them, and a parameter set is output according to the fused sequence, the word sequence includes a vector corresponding to at least one word in an input corpus, the action sequence includes a vector corresponding to at least one action, the at least one action includes an action corresponding to the at least one word, the at least one action is a sign language action, the parameter set includes action parameters, the action parameters are used to render an action image, the action image includes an image of a sign language action, and the sign language action is used to express the meaning of the input corpus.

16. The method according to claim 15, characterized in that The neural network is specifically used to fuse the word sequence, the action sequence and the expression sequence to obtain the fused sequence, wherein the expression sequence is obtained according to the input corpus, and the expression sequence includes a vector corresponding to at least one expression; The parameter set also includes expression parameters, and the expression parameters are used to render an image of the expression corresponding to the input corpus.

17. A computing device, characterized in that The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

18. A training device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 15 to 16 is implemented.

19. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method according to any one of claims 1 to 7 or 15 to 16.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 or 15 to 16 is implemented.

Images