Method for generating face image based on mouth mold, training method for model and device

Abstract

To provide a method for generating a face image based on a mouth mold, a training method for a model and a device.SOLUTION: The disclosure relates to a field of artificial intelligence, especially, fields of cloud computing and digital human. In details, recognition object audio data and a preset face image, then an audio feature of the recognition object audio data, the audio feature includes a speaking speed feature and semantic feature. On the basis of the speaking speed feature and semantic feature, processing is executed to the preset face image, for forming a face image having a mouth mold. By combining the semantic feature and speaking speed feature of the audio data, it is possible to support correct driving of the mouth mold in the face image at any speaking speed, and improve determination accuracy of the face image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of cloud computing and digital humans in the field of artificial intelligence, and in particular to a method for generating a face image based on mouth shape, a method and a device for training a model. [Background technology]

[0002] With the rapid development of artificial intelligence technology, digital human applications have become the mainstream of current research. The face of a digital human can be changed by voice, for example, the facial expression and mouth shape in the face image of a digital human can be changed by voice.

[0003] One of the core technologies in digital human applications is the technology of driving the mouth shape of a face with audio, and how to accurately match the mouth shape in a face image with audio data is a technical challenge that needs to be solved urgently. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a method for generating a face image based on mouth shapes, a method and a device for training a model. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a method for generating a face image based on a mouth shape, the method for generating a face image based on a mouth shape comprising: acquiring audio data to be recognized and a preset face image; determining audio features of the recognition target audio data, the audio features including speech rate features and semantic features; and processing the predetermined face image based on the speech rate feature and the semantic feature to generate a face image having a mouth shape.

[0006] According to a second aspect of the present disclosure, there is provided a method for training a face and mouth typing model, the method including: acquiring training target image data and a preset facial image, the training target image data including training target audio data and a training target facial image, the training target facial image having a mouth shape corresponding to the training target audio data; determining audio features of the training subject audio data, the audio features including speaking rate features and semantic features; training an initial face and mouth shape determination model based on the speech rate feature, the semantic feature and the preset face image to obtain a face image having a mouth shape; and determining that a training-completed face and mouth shape determination model is obtained if the face image having the mouth shape and the training target face image match.

[0007] According to a third aspect of the present disclosure, there is provided an apparatus for generating a face image based on a mouth shape, the apparatus for generating a face image based on a mouth shape comprising: A data acquisition unit for acquiring recognition target audio data and a preset face image; A feature determining unit used for determining audio features of the recognition target audio data, the audio features including speech rate features and semantic features; an image generating unit, which is used for processing the preset face image based on the speech rate feature and the semantic feature to generate a face image having a mouth shape.

[0008] According to a fourth aspect of the present disclosure, there is provided an apparatus for training a face and mouth typing model, the apparatus comprising: an image acquisition unit for acquiring training target image data and a preset face image, the training target image data including training target audio data and a training target face image, and the training target face image has a mouth shape corresponding to the training target audio data; A feature extraction unit used for determining audio features of the training target audio data, the audio features including speech rate features and semantic features; a model training unit, which is used for training an initial face and mouth shape determination model according to the speech rate feature, the semantic feature and the preset face image to obtain a face image with a mouth shape; and a model obtaining unit, which is used for determining that a face image having a mouth shape and the training target face image are consistent with each other, to obtain a face mouth shape determination model that has been completed training.

[0009] According to a fifth aspect of the present disclosure, there is provided an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to perform the methods described in the first and second aspects of the present disclosure.

[0010] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, ,Ko The computer instructions are for causing the computer to carry out the methods according to the first and second aspects of the present disclosure.

[0011] According to a seventh aspect of the present disclosure, a computer program M Provide ,beforeWhen the computer program is executed by a processor, the methods according to the first and second aspects of the present disclosure are realized.

[0012] The technology disclosed herein improves the accuracy of generating face images based on mouth shapes.

[0013] It should be understood that the contents described herein are not intended to determine key or critical features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood from the following specification. [Brief description of the drawings]

[0014] The drawings are used for a better understanding of the present invention and are not intended to limit the present disclosure. [Figure 1] 1 is a flowchart of a method for generating a face image based on a mouth shape provided by an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of a method for generating a face image based on a mouth shape provided by an embodiment of the present disclosure. [Diagram 3] 1 is a flowchart of a method for generating a face image based on a mouth shape provided by an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a method for training a face and mouth type determination model provided by an embodiment of the present disclosure. [Diagram 5] 1 is a flowchart of a method for training a face and mouth type determination model provided by an embodiment of the present disclosure. [Figure 6] FIG. 2 is a configuration diagram of an apparatus for generating a face image based on a mouth shape provided by an embodiment of the present disclosure. [Figure 7] FIG. 2 is a configuration diagram of an apparatus for generating a face image based on a mouth shape provided by an embodiment of the present disclosure. [Figure 8] FIG. 2 is a configuration diagram of an apparatus for training a face and mouth type determination model provided by an embodiment of the present disclosure. [Figure 9]FIG. 1 is a block diagram of an electronic device for implementing a method for generating a face image based on mouth shape and a method for training a model according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram of an electronic device for implementing a method for generating a face image based on mouth shape and a method for training a model according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings, and various details of the embodiments of the present disclosure are included for ease of understanding, but they should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, in the following description, for clarity and conciseness, descriptions of well-known functions and structures are omitted.

[0016] In current digital human applications, a core technology is to drive the mouth shape of a face with audio, that is, to change the mouth shape in a face image through audio data, and to match the mouth shape in a face image with the audio data. Therefore, how to drive a more realistic and accurate face mouth shape is a technical problem that needs to be solved as soon as possible.

[0017] Related In the method of generating a facial image based on mouth shapes, it is difficult to handle changes in speaking speed, and the speaking speed of audio data has a large impact on the mouth shapes. When the same sentence is spoken at different speaking speeds, the corresponding mouth shapes may be completely different. When the speaking speed is slow, the mouth shapes of each character can be perfectly aligned with the pronunciation. However, when the speaking speed is fast, the mouth shapes in the facial image do not accelerate at an equal rate, and if one mouth shape is not completed in time, the pronunciation of the next character may be required. This causes the mouth shapes of many characters to change, and various phenomena such as "character omission" and "continuous reading" appear, and many mouth shapes are lost, merged, or simplified, which affects the accuracy of generating a facial image.

[0018] The present disclosure provides a method for generating a facial image based on mouth shape, a method and a device for training a model, which are applied in the fields of cloud computing and digital human in the field of artificial intelligence, to improve the accuracy of generating a facial image with mouth shape.

[0019] It should be noted that the model in this example is not targeted to any specific user and does not reflect any personal information of any specific user. Furthermore, the facial images in this example are from a public dataset.

[0020] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and other processing of relevant users' personal information shall all conform to the provisions of relevant laws and regulations and not violate public order and morals.

[0021] In order to allow the reader to more deeply understand the implementation principles of the present disclosure, the embodiments will be further broken down with reference to the following FIGS.

[0022] 1 is a flowchart of a method for generating a face image based on a mouth shape provided by an embodiment of the present disclosure, which can be performed by an apparatus for generating a face image based on a mouth shape. As shown in FIG. 1, the method includes the following steps:

[0023] S101, audio data to be recognized and a preset face image are obtained.

[0024] Exemplarily, the face of the digital human may be designed in advance, for example, the shape of the face, eyes, nose, mouth, etc. of the digital human may be designed, and a preset face image may be generated. The digital human may change the mouth shape based on the preset face image, for example, in the preset face image, the mouth of the digital human may be closed, and the mouth shape of the digital human may change with the transmission of audio data.

[0025] The recognition target audio data is prepared in advance, and in the face image of the digital human, the mouth shape needs to change according to the recognition target audio data. Pre-set recognition target audio data and a pre-set face image are obtained. The recognition target audio data is an audio stream, and the pre-set face image may be a two-dimensional or three-dimensional image.

[0026] S102, determining audio features of the recognition target audio data, where the audio features include speech rate features and semantic features.

[0027] For example, after obtaining the audio data to be recognized, feature extraction is performed on the audio data to be recognized to obtain audio features of the audio data to be recognized. The audio features may include speech rate features and semantic features. The speech rate features may be used to represent the change speed of phonemes in the audio data to be recognized, for example, the speech rate features may be represented as the number of phonemes output within one second, i.e., the number of phonemes in the audio data to be recognized and the time of the audio data to be recognized may be determined; Recognize the number of phonemes and the duration of the audio data In this embodiment, the speech rate characteristic of the recognition target audio data may be determined as an average speech rate characteristic of the recognition target audio data, or speech rate characteristics corresponding to different phonemes of the recognition target audio data may be determined.

[0028] The semantic features can be used to represent the meaning expressed by the phonemes in the audio data to be recognized. The audio data to be recognized may include a plurality of phonemes, and the semantic features of each phoneme can be determined for the audio data to be recognized. That is, the audio data to be recognized is divided into phonemes to obtain each phoneme in the audio data to be recognized, and semantic recognition can be performed on the phonemes to determine the semantic features. For example, the semantic recognition can be performed using a preset semantic recognition model, which can be a neural network model. It is also possible to preset an association relationship between the phonemes and the meaning, and search for the semantic features of each phoneme in the audio data to be recognized as the semantic features of the audio data to be recognized based on the preset association relationship.

[0029] S103, a predetermined face image is processed based on the speech rate feature and the semantic feature to generate a face image having a mouth shape.

[0030] For example, after obtaining the speech rate feature and the semantic feature, a process is performed on a preset face image based on the speech rate feature and the semantic feature, and a change in the mouth shape in the preset face image is controlled to obtain a face image having a mouth shape. For example, if the sound emitted by the recognition target audio data is "a", the mouth shape in the face image will be the mouth shape of "a". In this embodiment, the mouth shape in the face image is determined based on the semantic feature and the speech rate feature, and multiple face images corresponding to the recognition target audio data can be obtained. A face video of the recognition target audio data can also be determined based on the multiple face images.

[0031] The association relationship between the mouth shape and the speech rate feature, and the association relationship between the mouth shape and the semantic feature may be set in advance, or the association relationship between the mouth shape, the speech rate feature, and the semantic feature may be set in advance. Based on the association relationship set in advance, the mouth shape corresponding to the speech rate feature and the semantic feature is determined, and a face image having the mouth shape is generated. A neural network model for determining the mouth shape may be trained in advance, and the speech rate feature and the semantic feature are input as input data to this neural network model, and a face image having the mouth shape is output.

[0032] In this embodiment, the method further includes, when it is determined that the numerical value represented by the speaking rate feature of the recognition target audio data is smaller than a predetermined speaking rate threshold, performing processing on the predetermined facial image based on the semantic feature to generate a facial image having a mouth shape.

[0033] Specifically, when speaking at a slower rate, the mouth shapes of each character can be perfectly aligned with the pronunciation, but when speaking at a faster rate, the mouth shape may not be completed in time to pronounce the next character, resulting in many missing, merging, or abbreviated mouth shapes.

[0034] The speech rate threshold is preset, and after the speech rate feature is obtained, the value represented by the speech rate feature can be compared with the preset speech rate threshold. If it is determined that the numerical value represented by the speech rate feature of the recognition target audio data is equal to or greater than the preset speech rate threshold, it indicates that the speech rate is fast, and a predetermined facial image can be processed based on the speech rate feature and the semantic feature to generate a facial image with a mouth shape.

[0035] When it is determined that the numerical value represented by the speech rate feature of the recognition target audio data is smaller than the preset speech rate threshold, it is determined that the speech rate of the recognition target audio data is slow, and a facial image having a mouth shape can be generated by processing a preset facial image using only the semantic features. For example, the semantic features alone are used as input data for a preset neural network model, and processing such as convolution of the semantic features can be performed, thereby reducing the amount of calculation required for processing a facial image.

[0036] The beneficial effect of such a setting is that when the speech rate of the audio data to be recognized is slow, accurate mouth shapes can be obtained based on semantic features alone, the amount of calculations can be reduced, and the efficiency of generating facial images can be improved.

[0037] In an embodiment of the present disclosure, audio data to be recognized is acquired, and speech rate features and semantic features are determined from the audio data to be recognized. The speech rate features and semantic features are combined to perform processing on a preset facial image. Here, the preset facial image is an initial image that serves as a basis when the mouth shape changes, and can represent the appearance of the face. Based on the speech rate features and semantic features, facial images with different mouth shapes are generated, and the mouth shapes of the facial images are matched with the audio data to be recognized. The problem of missing characters and continuous reading occurring in the mouth shapes of the facial images when the speaking speed is fast is solved. Accurate driving of the mouth shapes in the facial images is realized, and the accuracy of determining the facial images is improved.

[0038] FIG. 2 is a flowchart of a method for generating a face image based on a mouth shape provided by an embodiment of the present disclosure, which is an alternative embodiment based on the above embodiment.

[0039] In this embodiment, determining audio features of the audio data to be recognized can be subdivided as follows: determining speaking rate features of the audio data to be recognized based on a predetermined first feature extraction model, which is used for extracting the speaking rate features from the audio data to be recognized; determining semantic features of the audio data to be recognized based on a predetermined second feature extraction model, which is used for extracting the semantic features from the audio data to be recognized.

[0040] As shown in FIG. 2, the method includes the following steps.

[0041] S201, audio data to be recognized and a preset face image are acquired.

[0042] Illustratively, this step may refer to step S101 above, and will not be described further.

[0043] S202, determining a speaking rate feature of the recognition target audio data according to a preset first feature extraction model, where the first feature extraction model is used to extract the speaking rate feature from the recognition target audio data.

[0044] Exemplarily, a first feature extraction model is preset, which may be a neural network model predetermined to extract speaking rate features from the recognition target audio data. The recognition target audio data is input to the first feature extraction model and processed to obtain speaking rate features of the recognition target audio data. For example, the first feature extraction model may include network layers such as a convolution layer and a pooling layer, and the recognition target audio data is subjected to convolution processing and feature extraction to obtain speaking rate features of the recognition target audio data. In this embodiment, the network structure of the first feature extraction model is not particularly limited.

[0045] In this embodiment, determining a speaking rate feature of the recognition-target audio data based on a predetermined first feature extraction model includes: inputting the recognition-target audio data into the predetermined first feature extraction model to extract features and obtain a speech posterior probability feature of the recognition-target audio data, where the speech posterior probability feature represents information of a phoneme category of the recognition-target audio data; and determining a speaking rate feature of the recognition-target audio data based on the speech posterior probability feature of the recognition-target audio data.

[0046] Specifically, the first feature extraction model may be an ASR (Automatic Speech Recognition) model, and the ASR model may include multiple network layers, for example, a convolutional layer, a pooling layer, and a fully connected layer. The recognition target audio data is input to a predetermined ASR model to perform feature extraction, for example, feature extraction is performed by a convolutional layer, and PPG (Phonetic Posterioram) features of the recognition target audio data can be obtained. The PPG feature is a matrix of time to category that can represent the posterior probability of each phonetic category in each specific time frame of one utterance. The PPG feature can be represented using an image of a two-dimensional coordinate axis, which represents the information of the phoneme category of the recognition target audio data, where the horizontal axis represents time and the vertical axis represents the phoneme category.

[0047] After obtaining the PPG features, calculations can be performed on the PPG features based on a preset speaking rate determination algorithm, and the PPG features can be converted into speaking rate features of the target audio data. The phoneme change rate can be calculated and the speaking rate can be expressed as the speaking rate, thereby realizing explicit modeling of the speaking rate features. In this embodiment, the preset speaking rate determination algorithm is not particularly limited.

[0048] The beneficial effect of this setting is that the target audio data is input to the automatic speech recognition model for processing to obtain the PPG features of the target audio data, and further calculations are performed on the PPG features to obtain the speaking rate features, which realizes explicit modeling of the speaking rate, thereby introducing the speaking rate features, and greatly improves the accuracy and authenticity of the audio-driven mouth shape when the speaking rate changes.

[0049] In this embodiment, determining a speaking rate feature of the recognition-target audio data based on the speech posterior probability feature of the recognition-target audio data includes: performing a fast Fourier transform process on the speech posterior probability feature to obtain a frequency domain signal feature, where the frequency domain signal feature represents information of a phoneme category of the recognition-target audio data; dividing the frequency domain signal feature into frequency domain signal features of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain a speaking rate feature of the recognition-target audio data.

[0050] Specifically, the PPG features are time domain signals, and after obtaining the PPG features of the target audio data, the PPG features can be subjected to a fast Fourier transform process. That is, the PPG features are transformed into the frequency domain by FFT (Fast Fourier Transform) to obtain frequency domain signal features corresponding to the PPG features. The frequency domain signal features may be represented as information on the phoneme categories of the target audio data.

[0051] The frequency domain signal feature is integrated for each frequency band, and a desired frequency is calculated as the speaking rate size, i.e., the speaking rate feature of the audio data to be recognized is obtained. When calculating the speaking rate feature, the frequency band size can be set in advance, and the frequency domain signal feature is divided based on the preset frequency band size to obtain frequency domain signal features of multiple frequency band sizes. An integration process is performed on the frequency domain signal feature of each frequency band size one by one, and the integration result can be used as an embodiment of the phoneme change rate in the audio data to be recognized, i.e., the speaking rate feature.

[0052] The beneficial effect of such a setting is that by performing FFT processing and integration calculations, the PPG features can be converted into a specific speaking rate magnitude, and the speech rate feature can be determined, thereby improving the accuracy of generating face images.

[0053] S203, determining semantic features of the recognition target audio data based on a preset second feature extraction model, where the second feature extraction model is used to extract the semantic features from the recognition target audio data.

[0054] Exemplarily, the second feature extraction model may be a pre-trained neural network model, for example, the second feature extraction model is a pre-set semantic recognition model. The second feature extraction model includes a feature extraction network, and extracts semantic features from the recognition target audio data based on the pre-set second feature extraction model, thereby obtaining the semantic features of the recognition target audio data.

[0055] Through the first feature extraction model and the second feature extraction model, the speech rate features and semantic features can be quickly obtained, the separate extraction of the speech rate features and semantic features can be realized, the efficiency of feature extraction can be improved, and the efficiency of facial image generation can be further improved.

[0056] In this embodiment, determining semantic features of the recognition-target audio data based on the preset second feature extraction model includes: inputting the recognition-target audio data to the preset second feature extraction model to extract features, and outputting the semantic features of the recognition-target audio data.

[0057] Specifically, the second feature extraction model may be a semantic recognition model, and the semantic recognition model may include a network layer such as a multi-layer convolution layer to form a feature extraction network. The recognition target audio data is input to a pre-set semantic recognition model for processing, and feature extraction is performed, for example, by a convolution layer, to obtain semantic features of the recognition target audio data. The recognition target audio data may be streaming data, and the extracted semantic features may be streaming features. In this embodiment, the model structure of the semantic recognition model is not particularly limited.

[0058] The beneficial effect of such a setting is to perform automatic extraction of semantic features for the input audio stream data, improve the efficiency and accuracy of semantic feature determination, and further improve the efficiency and accuracy of facial image generation.

[0059] S204, the preset face image is processed based on the speech rate feature and the semantic feature to generate a face image having a mouth shape.

[0060] Illustratively, this step may refer to step S103 above, and will not be described further.

[0061] In an embodiment of the present disclosure, audio data to be recognized is acquired, and speech rate features and semantic features are determined from the audio data to be recognized. The speech rate features and semantic features are combined to perform processing on a preset facial image. Here, the preset facial image is an initial image that serves as a basis when the mouth shape changes, and can represent the appearance of the face. Based on the speech rate features and semantic features, facial images with different mouth shapes are generated, and the mouth shapes of the facial images are matched with the audio data to be recognized. The problem of missing characters and continuous reading occurring in the mouth shapes of the facial images when the speaking speed is fast is solved. Accurate driving of the mouth shapes in the facial images is realized, and the accuracy of determining the facial images is improved.

[0062] FIG. 3 is a flowchart of a method for generating a face image based on a mouth shape provided by an embodiment of the present disclosure, which is an alternative embodiment based on the above embodiment.

[0063] In this embodiment, processing a preset facial image based on speech rate features and semantic features to generate a facial image with a mouth shape can be subdivided as follows: the speech rate features and semantic features are input to a preset face and mouth shape determination model for processing, and a facial image with a mouth shape is generated based on the processed result and the preset facial image.

[0064] As shown in FIG. 3, the method includes the following steps.

[0065] S301, audio data to be recognized and a preset face image are acquired.

[0066] Illustratively, this step may refer to step S101 above, and will not be described further.

[0067] S302, determining audio features of the recognition target audio data, the audio features including speech rate features and semantic features.

[0068] Illustratively, this step may refer to step S102 above, and will not be described further.

[0069] S303: The speech rate feature and the semantic feature are inputted into a preset face and mouth shape determination model for processing, and a face image having a mouth shape is generated based on the processing result and the preset face image.

[0070] For example, a face and mouth type determination model, which is a neural network model that can be used to output a face image having a mouth type, is constructed and trained in advance. The speech rate feature and the semantic feature are input as input data to a preset face and mouth type determination model for processing. After processing, the face and mouth type determination model can change the mouth type of the preset face image based on the processing result to obtain a face image having a mouth type. For example, the processing result determined by the face and mouth type determination model based on the speech rate feature and the semantic feature may be the size and shape information of the mouth type, and the preset face image can be rendered based on the determined size and shape information of the mouth type to generate a face image including the mouth type. By using the face and mouth type determination model, a face image can be obtained quickly, and the problem of the deterioration of the effect of the audio-driven face and mouth type due to the change in speech rate can be avoided by combining the speech rate feature and the semantic feature, and the efficiency and accuracy of generating the face image can be improved.

[0071] In this embodiment, inputting the speech rate feature and the semantic feature into a preset face and mouth type determination model for processing, and generating a face image having a mouth shape based on the processed result and the preset face image includes: performing a combination process on the speech rate feature and the semantic feature based on the preset face and mouth type determination model to obtain a combination feature of the recognition target audio data, where the combination feature represents the speech rate feature and the semantic feature; performing feature extraction on the combination feature based on a convolution layer in the preset face and mouth type determination model to obtain face driving parameters, where the face driving parameters are used to represent parameters required to drive a change in the mouth shape in the face image; and performing image rendering on the preset face image based on the face driving parameters to generate a face image having a mouth shape.

[0072] Specifically, the speech rate feature and the semantic feature are input into a preset face and mouth type determination model. Based on the face and mouth type determination model, a combination process can be performed on the speech rate feature and the semantic feature, for example, a matrix represented by the speech rate feature and a matrix represented by the semantic feature can be combined. The combined data is determined as a combined feature of the recognition target audio data. That is, the combined feature can represent the speech rate feature and the semantic feature.

[0073] A network layer such as a convolution layer is set in the face and mouth type determination model, and when the combined feature passes through the convolution layer of the face and mouth type determination model, feature extraction can be performed on the combined feature based on the convolution layer, and a face driving parameter can be calculated. The face driving parameter is a parameter required to drive the change of the mouth shape in the face image. For example, the face driving parameter may be position information or size information of a target frame including the mouth shape in the face image. After obtaining the face driving parameter, image rendering is performed on the preset face image, and the mouth shape in the preset face image is changed from the original closed shape to a shape corresponding to the face driving parameter, and a face image having a mouth shape is obtained. For some of the recognition target audio data, multiple face images having different mouth shapes can be generated.

[0074] The beneficial effect of such a setting is to combine the speaking rate features and the semantic features, obtain the parameters required to drive the face and mouth shape through the driving network of the face and mouth shape determination model, adapt the mouth shape in the generated face image to the audio data to be recognized, reduce the influence of the speaking rate on the mouth shape in the face image, and improve the efficiency and accuracy of generating face images.

[0075] In this embodiment, the facial driving parameters are weight parameters of a mixed deformation, and performing image rendering on a predetermined facial image based on the facial driving parameters to generate a facial image having a mouth shape includes determining facial 3D mesh data corresponding to the predetermined facial image based on the weight parameters of the mixed deformation, the facial 3D mesh data being data representing a 3D mesh model of the facial surface in the facial image, and performing image rendering on the predetermined facial image based on the facial 3D mesh data to generate a facial image having a mouth shape.

[0076] Specifically, the face driving parameters may be blend shape weights, and the blend shape weights are obtained by a driving network in a face mouth shape determination model. Based on the blend shape weight parameters, a preset rendering engine can obtain a face image with a mouth shape based on a preset face image. For example, the preset rendering engine can be an Unreal rendering engine.

[0077] When performing image rendering, 3D mesh data of the face can be determined first based on the blend shape weight. The 3D mesh data of the face can be data for expressing a 3D mesh model of the face surface in the face image. The 3D mesh of the face can be determined based on the blend shape weight and the blend shape base. Here, the blend shape base is related to the binding of the person image and is a fixed, invariant, predetermined parameter. After obtaining the 3D mesh data of the face, image rendering is performed on the face image to obtain a face image having a mouth shape.

[0078] The beneficial effect of such a setting is that we first obtain a 3D face mesh based on the blend shape weights, and then obtain a face image based on the 3D face mesh, which realizes accurate generation of face images and is convenient for users to experience digital humans.

[0079] In an embodiment of the present disclosure, audio data to be recognized is acquired, and speech rate features and semantic features are determined from the audio data to be recognized. The speech rate features and semantic features are combined to perform processing on a preset facial image. Here, the preset facial image is an initial image that serves as a basis when the mouth shape changes, and can represent the appearance of the face. Based on the speech rate features and semantic features, facial images with different mouth shapes are generated, and the mouth shapes of the facial images are matched with the audio data to be recognized. The problem of missing characters and continuous reading occurring in the mouth shapes of the facial images when the speaking speed is fast is solved. Accurate driving of the mouth shapes in the facial images is realized, and the accuracy of determining the facial images is improved.

[0080] FIG. 4 is a flowchart of a method for training a face and mouth typing model provided by an embodiment of the present disclosure, the method including: Model As shown in FIG. 4, the method includes the following steps.

[0081] S401, obtaining training target image data and a preset face image, the training target image data including training target audio data and a training target face image, and the training target face image has a mouth shape corresponding to the training target audio data.

[0082] Illustratively, a face mouth type determination model based on deep learning can be used when determining a face image having a mouth type. The face mouth type determination model can realize the method for generating a face image described in any of the above embodiments, and the face mouth type determination model needs to be trained in advance and then used. Obtain pre-collected training target image data and pre-set face images. The training target image data may include training target audio data and training target face images, where the training target audio data is an audio stream for training the model, and the training target face images have a mouth type that matches the training target audio data.

[0083] The preset face image is a face image of a digital human with a pre-designed mouth, and the preset face image may also include facial features such as eyes and nose. The face shape, eyes, nose, mouth, etc. of the digital human can be designed to generate a preset face image. The digital human can change the mouth shape based on the preset face image, for example, in the preset face image, the mouth of the digital human is in a closed state, and the mouth shape of the digital human can change with the transmission of audio data. The difference between the training target face image and the preset face image is that a change in the mouth shape occurs.

[0084] In this embodiment, obtaining training target image data includes obtaining training target audio data, performing a 3D reconstruction process of a face image based on the training target audio data to obtain 3D face mesh data corresponding to the training target audio data, and obtaining a training target face image based on the 3D face mesh data corresponding to the training target audio data.

[0085] Specifically, a pre-collected training set is obtained, and the training set may be training target audio data. A training target face image is generated based on the training target audio data. The training target face image has a mouth shape, and the mouth shape in the training target face image is compatible with the training target audio data.

[0086] A 3D reconstruction process of a face image can be performed based on the training target audio data. For example, a 3D reconstruction process of a face image of each frame is performed based on each phoneme of the training target audio data. In this embodiment, the processing procedure of the 3D reconstruction is not particularly limited. By determining 3D face mesh data for each frame, a 3D face mesh of multiple frames corresponding to the training target audio data can be obtained. A training target face image of multiple frames is obtained based on the 3D face mesh corresponding to the training target audio data.

[0087] A beneficial effect of such a setting is that predetermining a facial image corresponding to the training target audio data makes it easier to train the face and mouth type determination model, thereby improving the training efficiency and accuracy of the face and mouth type determination model.

[0088] S402, determining audio features of the training audio data, where the audio features include speech rate features and semantic features.

[0089] For example, after obtaining the training target audio data, feature extraction is performed on the training target audio data to obtain audio features of the training target audio data. The audio features may include speech rate features and semantic features. The speech rate features can be used to represent the change speed of phonemes in the training target audio data, for example, the speech rate features can be represented as the number of phonemes output within one second, that is, the number of phonemes in the training target audio data and the time of the training target audio data can be determined; The number of phonemes is calculated based on the duration of the training audio data. In this embodiment, the speaking rate characteristic of the training target audio data may be determined as an average speaking rate characteristic of the training target audio data, or the speaking rate characteristic corresponding to different phonemes of the training target audio data may be determined.

[0090] The semantic features can be used to represent the meaning expressed by the training target audio data. The training target audio data may include multiple phonemes, and the semantic features of each phoneme can be determined for the training target audio data. That is, the training target audio data can be divided into phonemes to obtain each phoneme in the training target audio data, and semantic recognition can be performed on the phonemes to determine the semantic features. For example, the semantic recognition can be performed using a preset semantic recognition model, which can be a neural network model. In addition, the association relationship between the phoneme and the meaning can be preset, and the semantic features of all the phonemes in the training target audio data can be searched for as the semantic features of the training target audio data based on the preset association relationship.

[0091] S403, training an initial face and mouth shape determination model based on the speech rate feature, the semantic feature and the preset face image, and obtaining a face image with a mouth shape.

[0092] For example, the speech rate feature and the semantic feature of the training audio data are input to the training face mouth type determination model for iterative training, and in each iteration, a face image with a mouth type is generated based on the processing result and a preset face image.

[0093] A training target face and mouth type determination model is constructed in advance, and speech rate features and semantic features are input as input data into the training target face and mouth type determination model for processing. After performing processing, the face and mouth type determination model can change the mouth shape of a preset face image based on the processing result to obtain a face image with a different mouth shape. For example, the processing result determined by the face and mouth type determination model based on the speech rate features and semantic features is mouth size and shape information, and a preset face image can be rendered based on the determined mouth size and shape information to generate a face image including the mouth shape. The training target audio data includes multiple phonemes, and a face image with a mouth shape corresponding to each phoneme can be generated.

[0094] S404, if the face image having the mouth shape and the training target face image are consistent, it is determined that a training completed face mouth shape determination model is obtained.

[0095] For example, after obtaining a facial image with a mouth shape output from the model, the facial image with a mouth shape corresponding to a phoneme is compared with the training target facial image corresponding to the phoneme. If the two are consistent, it is determined that the training of the face and mouth type determination model is completed. If the two are not consistent, it is determined that the face and mouth type determination model needs to be trained. Then, the semantic features and speaking rate features of the training target audio data are input into the face and mouth type determination model, and training is performed based on a preset back propagation algorithm until the facial image with the output mouth shape is consistent with the corresponding training target facial image.

[0096] In addition, a similarity threshold may be preset, and the similarity threshold can be used to determine whether the training of the face and mouth type determination model is completed. After obtaining a face image with a mouth type, a similarity between the face image with the mouth type and a corresponding training target face image is determined. If the determined similarity is equal to or greater than the preset similarity threshold, it is determined that the training of the face and mouth type determination model is completed, and if the similarity is smaller than the preset similarity threshold, it is determined that the training of the face and mouth type determination model is not completed.

[0097] In the embodiment of the present disclosure, training target audio data and training target facial images are obtained, and speaking rate features and semantic features are determined from the training target audio data. The speaking rate features and semantic features are combined to train a training target face mouth type determination model. Based on the speaking rate features and semantic features, facial images with different mouth shapes are generated, and the mouth shapes in the output facial images are matched with the training target audio data by training. The model learns the effects of different speaking rates on the mouth shapes, which greatly improves the accuracy and authenticity of the audio-driven mouth shapes when the speaking rate changes, and is convenient for improving the accuracy of facial image determination when the face mouth type determination model is used in the future.

[0098] FIG. 5 is a flowchart of a method for training a face and mouth type determination model provided by an embodiment of the present disclosure, which is an optional embodiment based on the above embodiment.

[0099] In this embodiment, determining audio features of the training target audio data can be subdivided as follows: determining speaking rate features of the training target audio data based on a preset first feature extraction model, where the first feature extraction model is used for extracting the speaking rate features from the training target audio data; determining semantic features of the training target audio data based on a preset second feature extraction model, where the second feature extraction model is used for extracting the semantic features from the training target audio data.

[0100] As shown in FIG. 5, the method includes the following steps.

[0101] S501, obtaining training target image data and a preset face image, the training target image data including training target audio data and a training target face image, and the training target face image has a mouth shape corresponding to the training target audio data.

[0102] Illustratively, this step can refer to step S401 above, and will not be described further.

[0103] S502, determining a speaking rate feature of the training target audio data based on a preset first feature extraction model, where the first feature extraction model is used to extract the speaking rate feature from the training target audio data.

[0104] Exemplarily, a first feature extraction model is preset, which may be a neural network model predetermined to extract speaking rate features from training target audio data. The training target audio data is input to the first feature extraction model and processed to obtain speaking rate features of the training target audio data. For example, the first feature extraction model may include network layers such as a convolution layer and a pooling layer, and the training target audio data is subjected to convolution processing and feature extraction to obtain speaking rate features of the training target audio data. In this embodiment, the network structure of the first feature extraction model is not particularly limited.

[0105] In this embodiment, determining a speaking rate feature of the training-target audio data based on a preset first feature extraction model includes: inputting the training-target audio data into the preset first feature extraction model to perform feature extraction, and obtaining a speech posterior probability feature of the training-target audio data, where the speech posterior probability feature represents information of a phoneme category of the training-target audio data; and determining a speaking rate feature of the training-target audio data based on the speech posterior probability feature of the training-target audio data.

[0106] Specifically, the first feature extraction model may be an ASR model, and the ASR model may include multiple network layers, for example, a convolutional layer, a pooling layer, and a fully connected layer. The training target audio data is input to a predetermined ASR model to perform feature extraction, for example, feature extraction is performed by a convolutional layer, and the PPG feature of the training target audio data can be obtained. The PPG feature is a matrix of time to category that can represent the posterior probability of each speech category in each specific time frame of an utterance. The PPG feature can be represented using a two-dimensional coordinate axis image, which represents the information of the phoneme category of the training target audio data, where the horizontal axis represents time and the vertical axis represents the phoneme category.

[0107] After obtaining the PPG features, calculations can be performed on the PPG features based on a preset speaking rate determination algorithm, and the PPG features can be converted into speaking rate features of the training target audio data. The phoneme change rate can be calculated and the speaking rate can be expressed as the speaking rate, thereby realizing explicit modeling of the speaking rate features. In this embodiment, the preset speaking rate determination algorithm is not particularly limited.

[0108] The beneficial effect of this setting is that the training target audio data is input to the automatic speech recognition model for processing to obtain the PPG features of the training target audio data, and further calculations are performed on the PPG features to obtain the speaking rate features, which realizes explicit modeling of the speaking rate, thereby introducing the speaking rate features, and greatly improves the accuracy and authenticity of the audio-driven mouth shape when the speaking rate changes.

[0109] In this embodiment, determining a speaking rate feature of the training target audio data based on the speech posterior probability feature of the training target audio data includes: performing a fast Fourier transform process on the speech posterior probability feature to obtain a frequency domain signal feature, where the frequency domain signal feature represents information of the phoneme category of the training target audio data; dividing the frequency domain signal feature into frequency domain signal features of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain the speaking rate feature of the training target audio data.

[0110] Specifically, the PPG features are time domain signals, and after obtaining the PPG features of the training target audio data, the PPG features can be subjected to a fast Fourier transform process, that is, the PPG features are transformed into the frequency domain by FFT, and the frequency domain signal features corresponding to the PPG features are obtained. The frequency domain signal features can be represented as the phoneme category information of the training target audio data.

[0111] The frequency domain signal feature is integrated for each frequency band, and the desired frequency is calculated as the speaking rate size, that is, the speaking rate feature of the training target audio data is obtained. When calculating the speaking rate feature, the frequency band size can be preset, and the frequency domain signal feature is divided based on the preset frequency band size to obtain frequency domain signal features of multiple frequency band sizes. An integration process is performed on the frequency domain signal feature of each frequency band size one by one, and the integration result can be used as an embodiment of the phoneme change rate in the training target audio data, that is, the speaking rate feature.

[0112] The beneficial effect of this setting is that by performing FFT processing and integral calculation, the PPG features can be converted into a specific speaking rate magnitude, and the determination of the speaking rate features can be realized, thereby improving the training accuracy of the face and mouth type determination model.

[0113] S503, determining semantic features of the training object audio data based on a preset second feature extraction model, where the second feature extraction model is used to extract the semantic features from the training object audio data.

[0114] Illustratively, the second feature extraction model may be a pre-trained neural network model, for example, the second feature extraction model is a pre-set semantic recognition model. The second feature extraction model includes a feature extraction network, and extracts semantic features from the training target audio data based on the feature extraction network in the second feature extraction model, thereby obtaining the semantic features of the training target audio data.

[0115] Through the first feature extraction model and the second feature extraction model, the speaking rate features and the semantic features can be quickly obtained, the separate extraction of the speaking rate features and the semantic features is realized, the efficiency of the feature extraction is improved, and the training efficiency of the face and mouth type determination model is further improved.

[0116] In this embodiment, determining the semantic features of the training target audio data based on the preset second feature extraction model includes: inputting the training target audio data into the preset second feature extraction model to extract features, and outputting the semantic features of the training target audio data.

[0117] Specifically, the second feature extraction model may be a semantic recognition model, and the semantic recognition model may include a network layer such as a multi-layer convolution layer to form a feature extraction network. The training target audio data may be input to a pre-set semantic recognition model for processing, and for example, feature extraction may be performed by a convolution layer to obtain semantic features of the training target audio data. The training target audio data may be streaming data, and the extracted semantic features may be streaming features. In this embodiment, the model structure of the semantic recognition model is not particularly limited.

[0118] The beneficial effect of such a setting is to automatically extract semantic features from the input audio stream data, improve the efficiency and accuracy of semantic feature determination, and further improve the efficiency and accuracy of training the face-mouth type determination model.

[0119] S504, training an initial face and mouth shape determination model based on the speech rate feature, the semantic feature and the preset face image, and obtaining a face image with a mouth shape.

[0120] For example, the speech rate feature and the semantic feature are input to a training target face and mouth type determination model for training. The training target face and mouth type determination model processes the semantic feature and the speech rate feature, and generates a face image with a mouth type based on the processing result and a preset face image.

[0121] In this embodiment, training an initial face and mouth type determination model based on a speech rate feature, a semantic feature, and a preset face image to obtain a face image with a mouth shape includes: performing a combination process on the speech rate feature and the semantic feature based on the initial face and mouth type determination model to obtain a combination feature of the training target audio data, where the combination feature represents the speech rate feature and the semantic feature; performing feature extraction on the combination feature based on a convolution layer in the initial face and mouth type determination model to obtain face driving parameters, where the face driving parameters are used to represent parameters required to drive a change in the mouth shape in the face image; and performing image rendering on the preset face image based on the face driving parameters to obtain a face image with a mouth shape.

[0122] Specifically, the speech rate feature and the semantic feature are input into a face and mouth type determination model to be trained. Based on the face and mouth type determination model, a combination process can be performed on the speech rate feature and the semantic feature, for example, a matrix represented by the speech rate feature and a matrix represented by the semantic feature can be combined. The combined data is determined as a combined feature of the training target audio data. That is, the combined feature can represent the speech rate feature and the semantic feature.

[0123] A network layer such as a convolution layer is set in the face mouth type determination model, and when the combined feature passes through the convolution layer of the face mouth type determination model, feature extraction is performed on the combined feature based on the convolution layer, and a face driving parameter can be calculated and obtained. The face driving parameter is a parameter required to drive the change of the mouth shape in the face image. For example, the face driving parameter may be position information or size information of a target frame including the mouth shape in the face image. After obtaining the face driving parameter, image rendering is performed on the preset face image, and the mouth shape in the preset face image is changed from the original closed shape to a shape corresponding to the face driving parameter, and a face image having a mouth shape is obtained.

[0124] The beneficial effect of such a setting is that by combining the speech rate features and the semantic features, and obtaining and training the parameters required to drive the face and mouth patterns through the driving network of the face and mouth pattern determination model, the mouth patterns in the generated face image are adapted to the training target audio data, the influence of the speech rate on the mouth patterns in the face image is reduced, and the training accuracy of the face and mouth pattern determination model is improved.

[0125] In this embodiment, the facial driving parameters are weight parameters of a mixed deformation, and performing image rendering on a predetermined facial image based on the facial driving parameters to obtain a facial image having a mouth shape includes determining facial 3D mesh data corresponding to the predetermined facial image based on the weight parameters of the mixed deformation, the facial 3D mesh data being data representing a 3D mesh model of the facial surface in the facial image, and performing image rendering on the predetermined facial image based on the facial 3D mesh data to generate a facial image having a mouth shape.

[0126] Specifically, the face driving parameters may be blend shape weights, and the blend shape weights are obtained by a driving network in a face mouth shape determination model. Based on the blend shape weight parameters, a preset rendering engine can obtain a face image with a mouth shape based on a preset face image. For example, the preset rendering engine can be an Unreal rendering engine.

[0127] When performing image rendering, 3D mesh data of the face can be determined first based on the blend shape weight. The 3D mesh data of the face can be data for representing a 3D mesh model of a face surface in a face image. The 3D mesh of the face can be determined based on the blend shape weight and the blend shape base. Here, the blend shape base is related to the binding of the person image and is a fixed, invariant, predetermined parameter. After obtaining the 3D mesh data of the face, image rendering is performed on the face image to obtain a face image having a mouth shape.

[0128] The beneficial effect of such a setting is to first obtain a 3D facial mesh based on the blend shape weights, and then obtain a facial image based on the 3D facial mesh, thereby realizing accurate generation of facial images and improving the training accuracy of the face-mouth type determination model.

[0129] S505, if the face image having the mouth shape and the training target face image are consistent, it is determined that a training completed face mouth shape determination model is obtained.

[0130] Illustratively, this step can refer to step S404 above, and will not be described further.

[0131] In the embodiment of the present disclosure, training target audio data and training target facial images are obtained, and speaking rate features and semantic features are determined from the training target audio data. The speaking rate features and semantic features are combined to train a training target face mouth type determination model. Facial images with different mouth shapes are generated based on the speaking rate features and semantic features, and the mouth shapes in the output facial images are matched with the training target audio data through training. The model learns the effects of different speaking rates on the mouth shapes, which greatly improves the accuracy and authenticity of the audio-driven mouth shapes when the speaking rate changes, and is convenient for improving the accuracy of facial image determination when the face mouth type determination model is used in the future.

[0132] 6 is a block diagram of an apparatus for generating a face image based on a mouth shape provided by an embodiment of the present disclosure. For ease of explanation, only parts related to the embodiment of the present disclosure are shown. Referring to FIG. 6, the apparatus for generating a face image based on a mouth shape 600 includes a data acquisition unit 601, a feature determination unit 602, and an image generation unit 603.

[0133] The data acquisition unit 601 is used for acquiring the target audio data and the preset face image; The feature determining unit 602 is used for determining audio features of the recognition target audio data, the audio features including speech rate features and semantic features; The image generating unit 603 is used for processing the pre-defined face image according to the speech rate feature and the semantic feature, and generating a face image with a mouth shape.

[0134] FIG. 7 is a configuration diagram of an apparatus for generating a facial image based on a mouth shape provided by an embodiment of the present disclosure. As shown in FIG. 7, the apparatus for generating a facial image based on a mouth shape 700 includes a data acquisition unit 701, a feature determination unit 702, and an image generation unit 703, where the feature determination unit 702 includes a first determination module 7021 and a second determination module 7022.

[0135] The first determination module 7021 is used for determining the speech rate feature of the recognition target audio data according to a first feature extraction model preset. , No. The feature extraction model 1 is used to extract speech rate features from the target audio data, The second determination module 7022 is used for determining semantic features of the recognition target audio data based on a preset second feature extraction model, and the second feature extraction model is used for extracting semantic features from the recognition target audio data.

[0136] In one example, the first determination module 7021 is a feature extraction sub-module for inputting the recognition target audio data into a preset first feature extraction model to perform feature extraction and obtain a phonetic posterior probability feature of the recognition target audio data, the phonetic posterior probability feature representing information of a phoneme category of the recognition target audio data; a feature determining sub-module, which is used for determining a speaking rate feature of the recognition target audio data based on the speech posterior probability feature of the recognition target audio data.

[0137] In one example, the feature determination submodule specifically includes: performing a fast Fourier transform process on the speech posterior probability features to obtain frequency domain signal features, the frequency domain signal features representing phoneme category information of the recognition target audio data; splitting the frequency domain signal characteristics into frequency domain signal characteristics of at least two frequency bands based on a predetermined frequency band size; and performing integration processing on the frequency domain signal features of the at least two frequency bands to obtain the speech rate feature of the recognition target audio data.

[0138] In one example, the second determination module 7022 specifically determines whether The recognition target audio data is input to a preset second feature extraction model to extract features, and semantic features of the recognition target audio data are obtained as an output.

[0139] In one example, the image generation unit 703 includes: and an image generation module, which is used to input the speech rate features and the semantic features into a preset face and mouth shape determination model for processing, and generate a face image having a mouth shape based on the processed result and the preset face image.

[0140] In one example, the image generation module includes: a feature combining submodule, which is used for performing combining processing on the speech rate feature and the semantic feature based on the preset face and mouth type determination model to obtain a combined feature of the recognition target audio data, the combined feature representing a speech rate feature and a semantic feature; a parameter determination submodule, which is used to perform feature extraction on the combined features based on a convolution layer in the preset face and mouth shape determination model to obtain face driving parameters, the face driving parameters being used to represent parameters required to drive changes in mouth shapes in face images; and an image rendering sub-module, which is used for performing image rendering on the preset face image according to the face driving parameters to generate a face image having a mouth shape.

[0141] In one example, the face driving parameters are weight parameters of a blended deformation, and the image rendering submodule specifically includes: determining 3D facial mesh data corresponding to the preset facial image based on weight parameters of the mixed transformation, the 3D facial mesh data being data representing a 3D mesh model of a facial surface in the facial image; and performing image rendering on the preset face image based on the face 3D mesh data to generate a face image having a mouth shape.

[0142] One example further includes: and a semantic processing unit, which is used to process the predetermined facial image based on the semantic features and generate a facial image having a mouth shape when it is determined that a numerical value represented by the speech rate feature of the recognition target audio data is smaller than a predetermined speech rate threshold.

[0143] 8 is a block diagram of an apparatus for training a face and mouth type determination model provided by an embodiment of the present disclosure. For ease of explanation, only parts related to the embodiment of the present disclosure are shown. Referring to FIG. 8, the apparatus for training a face and mouth type determination model 800 includes an image acquisition unit 801, a feature extraction unit 802, a model training unit 803, and a model acquisition unit 804.

[0144] The image acquisition unit 801 is used for acquiring training object image data and a preset face image, the training object image data includes training object audio data and a training object face image, and the training object face image has a mouth shape corresponding to the training object audio data; The feature extraction unit 802 is used for determining audio features of the training target audio data, the audio features including speech rate features and semantic features; The model training unit 803 is used for training an initial face and mouth shape determination model according to the speech rate feature, the semantic feature and the preset face image to obtain a face image with a mouth shape; The model obtaining unit 804 is used for determining that if the face image with mouth shape and the training target face image are consistent, a trained face mouth shape determination model is obtained.

[0145] In one example, the feature extraction unit 802 includes: A first extraction module used for determining a speaking rate feature of the training target audio data based on a first feature extraction model previously set, the first feature extraction model being used for extracting the speaking rate feature from the training target audio data; a second extraction module used for determining semantic features of the training target audio data based on a second feature extraction model preset, the second feature extraction model being used for extracting semantic features from the training target audio data.

[0146] In one example, the first extraction module comprises: a probability determination sub-module, which is used for inputting the training target audio data into a preset first feature extraction model to perform feature extraction and obtain a phonetic posterior probability feature of the training target audio data, the phonetic posterior probability feature representing information of a phoneme category of the training target audio data; a speaking rate determining sub-module, which is used for determining a speaking rate feature of the training target audio data according to the speech posterior probability feature of the training target audio data.

[0147] In one example, the speech rate determination submodule comprises: performing a fast Fourier transform process on the speech posterior probability features to obtain frequency domain signal features, the frequency domain signal features representing phoneme category information of the training audio data; splitting the frequency domain signal characteristics into frequency domain signal characteristics of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain the speaking rate feature of the training target audio data.

[0148] In one example, the second extraction module specifically comprises: The training object audio data is input to a preset second feature extraction model to extract features, and semantic features of the training object audio data are obtained as an output.

[0149] In one example, the model training unit 803 includes: a feature combination module, which is used for performing a combination process on the speech rate feature and the semantic feature based on the initial face and mouth type determination model to obtain a combination feature of the training target audio data, the combination feature representing a speech rate feature and a semantic feature; A parameter determination module is used to perform feature extraction on the combined features based on a convolution layer in the initial face and mouth shape determination model to obtain face driving parameters, the face driving parameters being used to represent parameters required to drive changes in mouth shapes in a face image; and an image rendering module for performing image rendering on the preset face image based on the face driving parameters to obtain a face image having a mouth shape.

[0150] In one example, the facial driving parameters are blended deformation weight parameters, and the image rendering module: a data determination sub-module, which is used for determining a facial 3D mesh data corresponding to the preset facial image based on a weight parameter of the mixed transformation, the facial 3D mesh data being data representing a 3D mesh model of a facial surface in the facial image; and an image rendering sub-module, which is used for performing image rendering on the preset face image according to the face 3D mesh data to generate a face image with a mouth shape.

[0151] In one example, the image acquisition unit 801 includes: a data acquisition module for acquiring the training target audio data; a 3D reconstruction module, which is used for performing 3D reconstruction processing of a face image according to the training target audio data, and obtaining 3D face mesh data corresponding to the training target audio data; and an image acquisition module, used for obtaining the training target face image based on the facial 3D mesh data corresponding to the training target audio data.

[0152] FIG. 9 is a block diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG. 9, the electronic device 900 includes at least one processor 902 and a memory 901 communicatively connected to the at least one processor 902, the memory storing instructions executable by the at least one processor 902, and the instructions, when executed by the at least one processor 902, cause the at least one processor 902 to execute the method for generating a facial image based on a mouth shape and the method for training a model of the present disclosure.

[0153] The electronic device 900 further includes a receiver 903 and a transmitter 904. The receiver 903 is used to receive commands and data sent from other devices, and the transmitter 904 is used to transmit commands and data to external devices.

[0154] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program. M provide.

[0155] According to an embodiment of the present disclosure, the present disclosure further comprises a computer program M Provides computer programs M is , stored in a readable storage medium R, At least one processor of the electronic device can read a computer program from a readable storage medium, and the computer program is executed by the at least one processor to cause the electronic device to execute the technical solution provided by any of the above embodiments.

[0156] 10 is a schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure. The electronic device may be a laptop computer, a desktop computer, a workbench, a personal digital assistant, or any other device. assistant The electronic devices are intended to represent various forms of digital computers, such as computers, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the practice of the present disclosure as described and / or claimed herein.

[0157] As shown in FIG. 10, the device 1000 includes a computing unit 1001, which has a read-only memory ( Read Only Memory, A computer program stored in the random access memory (ROM) 1002 or a storage unit 1008 Random Access Memory, The computing unit 1001, the ROM 1002, and the RAM 1003 can execute various appropriate operations and processes based on a computer program loaded into the RAM 1003. The RAM 1003 may further store various programs and data necessary for the operation of the device 1000. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other by a bus 1004. Input / Output, An I / O interface 1005 is also connected to the bus 1004 .

[0158] A number of components in the device 1000 are connected to an I / O interface 1005, including an input unit 1006, e.g. a keyboard, a mouse, etc., an output unit 1007, e.g. various types of displays, speakers, etc., a storage unit 1008, e.g. a magnetic disk, an optical disk, etc., and a communication unit 1009, e.g. a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 enables the device 1000 to exchange information / data with other devices via a computer network, e.g. the Internet, and / or various telecommunication networks.

[0159] The computing unit 1001 may be any of a variety of general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 include a central processing unit ( Central Processing Unit, CPU), Graphics Processing Unit ( Graphics Processing Unit, GPU, various dedicated artificial intelligence ( Artificial Intelligence, AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors ( Digital Signal Processor,The computing unit 1001 may include, but is not limited to, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the methods and processes described above, such as the method for generating a facial image based on a mouth shape and the method for training a model. For example, in some embodiments, the method for generating a facial image based on a mouth shape and the method for training a model are implemented as computer software programs and tangibly included in a machine-readable medium, such as the storage unit 1008. In some embodiments, a part or all of the computer programs may be loaded and / or installed in the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer programs are loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method for generating a facial image based on a mouth shape and the method for training a model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the method for generating a facial image based on a mouth shape and the method for training a model in any other suitable manner (e.g., by firmware).

[0160] Various embodiments of the systems and techniques described herein may be implemented using digital electronic circuit systems, integrated circuit systems, field programmable gate arrays ( Field-Programmable Gate Array, FPGA), dedicated integrated circuit ( Application Specific Integrated Circuit, ASIC), dedicated standard products ( Application Specific Standard Product, ASSP), System-on-chip system ( System On Chip, SOC), Complex Programmable Logic Device ( Complex Programmable Logic Device,The present invention may be implemented in a variety of applications, including a CPLD, computer hardware, firmware, software, and / or combinations thereof. In these various embodiments, the present invention may be embodied in one or more computer programs that can be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and that can receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0161] The program codes for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes can be executed entirely on the device, partially on the device, as a separate software package, partially on the device and partially on a remote device, or entirely on a remote device or server.

[0162] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program for use by or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (ROMs), and the like. Erasable Programmable Read Only Memory, EPROM ) or flash memory, optical fiber, portable compact disc read-only memory ( Compact Disc Read Only Memory, The content may include a CD-ROM, optical storage, magnetic storage, or any suitable combination of the above.

[0163] To provide for interaction with a user, the systems and techniques described herein can be implemented in a computer, which may include a display device (e.g., ,shadow polar ray tube (Cathode Ray Tube, CRT )also Liquid Crystal Display (Liquid Crystal Display, LCD ) monitor), and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the computer. Other types of devices may also be used to provide interaction with the user. For example, feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and may receive input from the user in any form (including acoustic input, speech input, or haptic input).

[0164] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user may interact with embodiments of the systems and techniques described herein), or a computing system including any combination of such back-end, middleware, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communications network). An example of a communications network is a local area network ( Local Area Networks, LAN), Wide Area Network ( Wide Area Networks, WAN) and the Internet.

[0165] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact with each other through a communication network. The relationship between the client and the server is generated by a computer program running on a corresponding computer and having a client-server relationship with each other. The server may be a cloud server, also called a cloud computing server or a cloud host, which is a host product of a cloud computing service system, thereby solving the defects of large management difficulty and weak service scalability that exist in traditional physical hosts and VPS services (abbreviated as "Virtual Private Server", or "VPS"). The server may be a server of a distributed system, or a server combined with blockchain.

[0166] As can be understood, the order of steps can be changed, added, or deleted using the various forms of flow shown above. For example, each step described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present application can be achieved, and is not limited to the present specification.

[0167] The above specific embodiments do not limit the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made based on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. 1. A method for generating a face image based on mouth shapes, comprising: acquiring audio data to be recognized and a preset face image; determining audio features of the recognition target audio data, the audio features including speech rate features and semantic features; and processing the predetermined face image based on the speech rate feature and the semantic feature to generate a face image having a mouth shape. A method for generating facial images based on mouth shapes.

2. Determining audio features of the recognition target audio data includes: determining a speech rate feature of the recognition target audio data based on a first feature extraction model that is preset, the first feature extraction model being used to extract the speech rate feature from the recognition target audio data; determining semantic features of the recognition-target audio data based on a second feature extraction model that is preset, the second feature extraction model being used to extract semantic features from the recognition-target audio data; The method of claim 1.

3. Determining a speech rate feature of the recognition target audio data based on a first feature extraction model set in advance, inputting the recognition target audio data into a preset first feature extraction model to extract features, thereby obtaining a phonetic posterior probability feature of the recognition target audio data, the phonetic posterior probability feature representing information of a phoneme category of the recognition target audio data; determining a speech rate feature of the recognition target audio data based on a speech posterior probability feature of the recognition target audio data; The method of claim 2.

4. Determining a speech rate feature of the recognition target audio data based on a speech posterior probability feature of the recognition target audio data, performing a fast Fourier transform process on the speech posterior probability features to obtain frequency domain signal features, the frequency domain signal features representing phoneme category information of the recognition target audio data; dividing the frequency domain signal characteristics into frequency domain signal characteristics of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain a speech rate feature of the recognition target audio data. The method according to claim 3.

5. determining semantic features of the recognition target audio data based on a second feature extraction model that is preset, inputting the recognition target audio data into a second feature extraction model set in advance to extract features, and outputting semantic features of the recognition target audio data; The method according to any one of claims 2 to 4.

6. performing processing on the predetermined face image based on the speech rate feature and the semantic feature to generate a face image having a mouth shape, inputting the speech rate feature and the semantic feature into a preset face and mouth shape determination model to process the speech rate feature and the semantic feature; and generating a face image having a mouth shape based on a result of the processing and the preset face image. The method according to any one of claims 1 to 5.

7. The speech rate feature and the semantic feature are inputted into a preset face and mouth shape determination model for processing, and a face image having a mouth shape is generated based on the processing result and the preset face image, performing a combination process on the speech rate feature and the semantic feature based on the preset face and mouth type determination model to obtain a combination feature of the recognition target audio data, the combination feature representing a speech rate feature and a semantic feature; Performing feature extraction on the combined features based on a convolutional layer in the preset face and mouth shape determination model to obtain face driving parameters, the face driving parameters being used to represent parameters required to drive changes in mouth shapes in a face image; and performing image rendering on the preset face image based on the face driving parameters to generate a face image having a mouth shape. The method according to claim 6.

8. The face driving parameters are weight parameters of a mixed deformation, and image rendering is performed on the preset face image based on the face driving parameters to generate a face image having a mouth shape, determining three-dimensional face mesh data corresponding to the preset face image based on weight parameters of the mixed transformation, the three-dimensional face mesh data being data representing a three-dimensional mesh model of a face surface in the face image; and performing image rendering on the preset face image based on the face 3D mesh data to generate a face image having a mouth shape. The method of claim 7.

9. and when it is determined that the numerical value represented by the speech rate feature of the recognition target audio data is smaller than a predetermined speech rate threshold, performing processing on the predetermined face image based on the semantic feature to generate a face image having a mouth shape. The method according to any one of claims 1 to 8.

10. 1. A method for training a face and mouth typing model, comprising: acquiring training target image data and a preset facial image, the training target image data including training target audio data and a training target facial image, the training target facial image having a mouth shape corresponding to the training target audio data; determining audio features of the training subject audio data, the audio features including speaking rate features and semantic features; training an initial face and mouth shape determination model based on the speech rate feature, the semantic feature and the preset face image to obtain a face image with a mouth shape; and determining that a training-completed face and mouth shape determination model is obtained when the face image having the mouth shape and the training target face image match. A method for training a face-mouth typing model.

11. Determining audio features of the training subject audio data includes: determining a speaking rate feature of the training target audio data based on a first feature extraction model that is preset, the first feature extraction model being used to extract the speaking rate feature from the training target audio data; determining semantic features of the training target audio data based on a second predetermined feature extraction model, the second feature extraction model being used to extract semantic features from the training target audio data; The method of claim 10.

12. Determining a speaking rate feature of the training target audio data based on a first feature extraction model that is preset, inputting the training target audio data into a preset first feature extraction model to perform feature extraction, and obtaining a phonetic posterior probability feature of the training target audio data, the phonetic posterior probability feature representing information of a phoneme category of the training target audio data; determining a speaking rate feature of the training target audio data based on speech posterior probability features of the training target audio data; The method of claim 11.

13. Determining a speaking rate feature of the training target audio data based on speech posterior probability features of the training target audio data includes: performing a fast Fourier transform process on the speech posterior probability features to obtain frequency domain signal features, the frequency domain signal features representing phoneme category information of the training audio data; dividing the frequency domain signal characteristics into frequency domain signal characteristics of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain a speech rate feature of the training target audio data. The method of claim 12.

14. determining semantic features of the training target audio data based on a second predetermined feature extraction model, inputting the training target audio data into a second feature extraction model set in advance to perform feature extraction, and outputting semantic features of the training target audio data; The method according to any one of claims 11 to 13.

15. Training an initial face and mouth shape determination model based on the speech rate feature, the semantic feature, and the preset face image to obtain a face image having a mouth shape, performing a combination process on the speech rate feature and the semantic feature based on the initial face and mouth type determination model to obtain a combination feature of the training target audio data, the combination feature representing a speech rate feature and a semantic feature; Performing feature extraction on the combined features based on a convolutional layer in the initial face and mouth shape determination model to obtain facial driving parameters, the facial driving parameters being used to represent parameters required to drive changes in mouth shapes in a facial image; and performing image rendering on the preset face image based on the face driving parameters to obtain a face image having a mouth shape. The method according to any one of claims 10 to 14.

16. The face driving parameters are weight parameters of a mixed deformation, and image rendering is performed on the preset face image based on the face driving parameters to obtain a face image having a mouth shape, determining three-dimensional face mesh data corresponding to the preset face image based on weight parameters of the mixed transformation, the three-dimensional face mesh data being data representing a three-dimensional mesh model of a face surface in the face image; and performing image rendering on the preset face image based on the face 3D mesh data to generate a face image having a mouth shape. The method of claim 15.

17. Obtaining training subject image data includes: obtaining said training subject audio data; performing a three-dimensional reconstruction process of a face image based on the training target audio data to obtain three-dimensional face mesh data corresponding to the training target audio data; obtaining the training target face image based on 3D face mesh data corresponding to the training target audio data; The method according to any one of claims 10 to 16.

18. An apparatus for generating a face image based on a mouth shape, comprising: A data acquisition unit is used for acquiring recognition target audio data and a preset face image; A feature determining unit used for determining audio features of the recognition target audio data, the audio features including a speech rate feature and a semantic feature; and an image generating unit for processing the predetermined face image based on the speech rate feature and the semantic feature to generate a face image having a mouth shape. An apparatus for generating facial images based on mouth shapes.

19. The characterization unit comprises: A first determination module used for determining a speech rate feature of the recognition target audio data based on a first feature extraction model set in advance, the first feature extraction model being used for extracting a speech rate feature from the recognition target audio data; a second determination module used for determining semantic features of the recognition target audio data based on a second predetermined feature extraction model, the second feature extraction model being used for extracting semantic features from the recognition target audio data; 20. The apparatus of claim 18.

20. The first determination module: a feature extraction submodule for inputting the recognition target audio data into a preset first feature extraction model to perform feature extraction and obtain a phonetic posterior probability feature of the recognition target audio data, the phonetic posterior probability feature representing information of a phoneme category of the recognition target audio data; a feature determining submodule, which is used for determining a speech rate feature of the recognition target audio data based on a speech posterior probability feature of the recognition target audio data; 20. The apparatus of claim 19.

21. The feature determination sub-module: performing a fast Fourier transform process on the speech posterior probability features to obtain frequency domain signal features, the frequency domain signal features representing phoneme category information of the recognition target audio data; dividing the frequency domain signal characteristics into frequency domain signal characteristics of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain a speech rate feature of the recognition target audio data.

21. The apparatus of claim 20.

22. The second determination module: The recognition target audio data is input to a second feature extraction model set in advance to extract features, and semantic features of the recognition target audio data are output. Apparatus according to any one of claims 19 to 21.

23. The image generating unit includes: and an image generation module for inputting the speech rate feature and the semantic feature into a preset face and mouth shape determination model for processing, and generating a face image having a mouth shape based on the processed result and the preset face image. Apparatus according to any one of claims 18 to 22.

24. The image generation module includes: a feature combining submodule, which is used for performing combining processing on the speech rate feature and the semantic feature based on the preset face and mouth type determination model to obtain a combined feature of the recognition target audio data, the combined feature representing a speech rate feature and a semantic feature; a parameter determination submodule, which is used to perform feature extraction on the combined features based on a convolution layer in the preset face and mouth shape determination model to obtain face driving parameters, the face driving parameters being used to represent parameters required to drive changes in mouth shapes in face images; and an image rendering submodule, which is used for performing image rendering on the preset face image according to the face driving parameters to generate a face image having a mouth shape.

24. The apparatus of claim 23.

25. The facial driving parameters are weight parameters of a blended transformation, and the image rendering submodule: determining three-dimensional face mesh data corresponding to the preset face image based on weight parameters of the mixed transformation, the three-dimensional face mesh data being data representing a three-dimensional mesh model of a face surface in the face image; and performing image rendering on the preset face image based on the face three-dimensional mesh data to generate a face image having a mouth shape.

24. The apparatus of claim 23.

26. a semantic processing unit, which is used to process the predetermined face image based on the semantic features when it is determined that the numerical value represented by the speech rate feature of the recognition target audio data is smaller than a predetermined speech rate threshold value, to generate a face image having a mouth shape; Apparatus according to any one of claims 18 to 25.

27. 1. An apparatus for training a face and mouth type determination model, comprising: an image acquisition unit for acquiring training target image data and a preset face image, the training target image data including training target audio data and a training target face image, and the training target face image has a mouth shape corresponding to the training target audio data; A feature extraction unit used for determining audio features of the training target audio data, the audio features including speech rate features and semantic features; a model training unit, which is used for training an initial face and mouth shape determination model according to the speech rate feature, the semantic feature and the preset face image to obtain a face image with a mouth shape; A model acquisition unit is used for determining that a face image having a mouth shape and the training target face image are consistent with each other, and a training completed face mouth shape determination model is obtained. A device for training face-mouth typing models.

28. The feature extraction unit comprises: A first extraction module used for determining a speaking rate feature of the training target audio data based on a preset first feature extraction model, the first feature extraction model being used for extracting the speaking rate feature from the training target audio data; a second extraction module used for determining semantic features of the training target audio data based on a preset second feature extraction model, the second feature extraction model being used for extracting semantic features from the training target audio data; 28. The apparatus of claim 27.

29. The first extraction module includes: a probability determination sub-module, which is used for inputting the training target audio data into a preset first feature extraction model to perform feature extraction and obtain a phonetic posterior probability feature of the training target audio data, the phonetic posterior probability feature representing information of a phoneme category of the training target audio data; a speaking rate determining submodule, which is used for determining a speaking rate characteristic of the training target audio data according to a speech posterior probability characteristic of the training target audio data; 30. The apparatus of claim 28.

30. The speech rate determination submodule: performing a fast Fourier transform process on the speech posterior probability features to obtain frequency domain signal features, the frequency domain signal features representing phoneme category information of the training audio data; dividing the frequency domain signal characteristics into frequency domain signal characteristics of at least two frequency bands based on a predetermined frequency band size; and performing an integration process on the frequency domain signal features of the at least two frequency bands to obtain a speech rate feature of the training target audio data.

30. The apparatus of claim 29.

31. The second extraction module includes: The training target audio data is input to a second feature extraction model set in advance to extract features, and semantic features of the training target audio data are output to obtain the semantic features. Apparatus according to any one of claims 28 to 30.

32. The model training unit includes: a feature combination module, which is used for performing a combination process on the speech rate feature and the semantic feature based on the initial face and mouth type determination model to obtain a combination feature of the training target audio data, the combination feature representing a speech rate feature and a semantic feature; A parameter determination module is used to perform feature extraction on the combined features based on a convolution layer in the initial face and mouth shape determination model to obtain face driving parameters, the face driving parameters being used to represent parameters required to drive changes in mouth shapes in a face image; and an image rendering module for performing image rendering on the preset face image based on the face driving parameters to obtain a face image having a mouth shape. Apparatus according to any one of claims 27 to 31.

33. The facial driving parameters are blended deformation weight parameters, and the image rendering module: a data determination submodule, which is used for determining a facial 3D mesh data corresponding to the preset facial image based on a weight parameter of the mixed transformation, the facial 3D mesh data being data representing a 3D mesh model of a facial surface in the facial image; and an image rendering submodule, which is used for performing image rendering on the preset face image based on the face three-dimensional mesh data to generate a face image having a mouth shape.

33. The apparatus of claim 32.

34. The image acquisition unit includes: a data acquisition module for acquiring the training target audio data; a 3D reconstruction module used for performing 3D reconstruction processing of a face image based on the training target audio data to obtain 3D face mesh data corresponding to the training target audio data; an image acquisition module adapted to obtain the training target face image based on face 3D mesh data corresponding to the training target audio data; Apparatus according to any one of claims 27 to 33.

35. An electronic device, At least one processor; and a memory communicatively connected to the at least one processor; An electronic device in which instructions executable by the at least one processor are stored in the memory, and the instructions are executed by the at least one processor to cause the at least one processor to execute a method according to any one of claims 1 to 9 or claims 10 to 17.

36. A non-transitory computer readable storage medium storing computer instructions, the computer instructions being for causing the computer to perform a method according to any one of claims 1 to 9 or claims 10 to 17.

37. A computer program product comprising a computer program, the steps of the method according to any one of claims 1 to 9 or claims 10 to 17 being realised when said computer program is executed by a processor.

Images