Bionic robot control method, device and equipment and storage medium

Abstract

The invention discloses a bionic robot control method and device, equipment and a storage medium, and relates to the technical field of bionic robots, and the method comprises the steps: obtaining a target character through a speech synthesis model, determining a phoneme sequence based on the target character, coding the phoneme sequence, and carrying out the preset convolution operation of the obtained coded sequence, a corresponding phoneme feature map is obtained; determining voice data based on the phoneme feature map through a voice synthesis model, and determining a voice time sequence based on the voice data; determining a target mouth shape based on the voice time sequence through a voice synthesis model, and determining a mouth shape time sequence based on the voice time sequence and the target mouth shape; and when the voice data is played, a motor of the mouth is controlled to perform corresponding motion based on the mouth shape time sequence, so that the bionic robot generates a corresponding mouth shape. Therefore, the action of the mouth of the bionic robot can be accurately controlled while the bionic robot plays the voice.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bionic robots, and in particular to a bionic robot control method, device, equipment and storage medium. BACKGROUND

[0002] In recent years, bionic humanoid robots are increasingly widely used in the fields of tourism and travel display, education, psychological treatment and the like. The naturalness and fluency of the bionic humanoid robots when interacting with users are crucial to improving user recognition. The mouth shape performance when the bionic head speaks has become a key indicator for measuring the interaction effect of the robot.

[0003] Currently, there are technical solutions for controlling bionic robots by performing time-frequency analysis and threshold segmentation on a voice file. However, such solutions can only realize the binary action of mouth opening and closing, and cannot complete fine control such as subtle changes in the lips, making it difficult to present a natural mouth shape. Another technical solution requires the use of at least three artificial intelligence models, including a large language model, a voice synthesis model, a mouth shape generation model and a voice-moouth shape alignment model. Not only does this solution require a large amount of labeled data for training, but it also has a high implementation difficulty. Moreover, the solution introduces obvious time delay in actual dialogue interaction due to multi-model processing, affecting the fluency of the interaction.

[0004] Therefore, how to accurately control the action of the mouth of a bionic robot while the bionic robot is playing a voice is a technical problem that needs to be solved at present. SUMMARY

[0005] Therefore, the present application aims to provide a bionic robot control method, device, equipment and storage medium, which can accurately control the action of the mouth of a bionic robot while the bionic robot is playing a voice. The specific solutions are as follows:

[0006] In a first aspect, the present application provides a bionic robot control method applied to a bionic robot comprising a voice synthesis model, comprising:

[0007] obtaining a target text through the voice synthesis model, determining a phoneme sequence based on the target text, encoding the phoneme sequence, and performing a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map;

[0008] determining voice data based on the phoneme feature map through the voice synthesis model, and determining a voice time sequence based on the voice data;

[0009] determining a target mouth shape based on the voice time sequence through the voice synthesis model, and determining a mouth shape time sequence based on the voice time sequence and the target mouth shape;

[0010] When the voice data is played, a motor of a mouth part is controlled based on the lip shape time sequence to make corresponding movement, so that the bionic robot makes corresponding lip shape.

[0011] Optionally, the determining the phoneme sequence based on the target text comprises:

[0012] When the target text contains Chinese, determining the pinyin of the Chinese as first phonemes;

[0013] When the target text contains English, determining the phonetic symbols of the English as second phonemes;

[0014] determining a phoneme sequence based on the first phonemes, the second phonemes and preset blank data; the preset blank data is data between single Chinese in the target text, between words, between the single Chinese and the words, and data of start and end positions of the target text.

[0015] Optionally, the encoding the phoneme sequence and performing preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map comprises:

[0016] transforming the phonemes in the phoneme sequence and the preset blank data based on a preset number set to complete the encoding operation of the phoneme sequence to obtain a corresponding encoded sequence; wherein the preset blank data is transformed into zero values in the preset number set in the encoding operation;

[0017] performing preset convolution operation on the encoded sequence through an embedding layer, a convolution layer and a neural network of the voice synthesis model to obtain a corresponding phoneme feature map.

[0018] Optionally, the determining voice data based on the phoneme feature map comprises:

[0019] determining first frame numbers corresponding to each phoneme in the phoneme feature map;

[0020] sampling the phoneme feature map based on the first frame numbers to obtain mel-frequency cepstral coefficients;

[0021] upsampling the mel-frequency cepstral coefficients to obtain corresponding voice data.

[0022] Optionally, the determining a voice time sequence based on the voice data comprises:

[0023] adding the first frame numbers corresponding to each phoneme in the voice data to obtain a second frame number of the voice data;

[0024] determine a speech time sequence of the speech data based on the second frame number, a preset sampling rate and a current time length of each frame; the preset sampling rate is a preset number of audio points collected in the speech data within a preset time period.

[0025] Optionally, the method further comprises:

[0026] determining each target phoneme in the speech time sequence;

[0027] selecting one target phoneme from the target phonemes as a current phoneme, and determining a first start time and a first end time of the current phoneme in the speech time sequence;

[0028] determining each target mouth shape corresponding to the current phoneme from a preset mouth shape set, and selecting one target mouth shape from the target mouth shapes as a current mouth shape;

[0029] determining a second start time and a second end time of the current mouth shape based on the first start time, the first end time, a number of the target mouth shapes and the current mouth shape;

[0030] jumping to the step of selecting one target mouth shape from the target mouth shapes as a current mouth shape until each target mouth shape corresponding to the current phoneme is selected;

[0031] jumping to the step of selecting one target phoneme from the target phonemes as a current phoneme until each target phoneme is selected;

[0032] when each target phoneme is selected, determining a corresponding mouth shape time sequence based on the second start time and the second end time of each target mouth shape of each target phoneme.

[0033] Optionally, the method further comprises:

[0034] in the speech time sequence, determining an end time of a previous phoneme adjacent to the current phoneme as a first start time of the current phoneme; wherein a start time of a first phoneme in the speech time sequence is set as a preset time;

[0035] determining the first end time of the current phoneme based on a current time length of each frame and the frame number of the current phoneme.

[0036] In a second aspect, the application provides a bionic robot control device applied to a bionic robot comprising a speech synthesis model, comprising:

[0037] a feature map determination module configured to acquire target text through the speech synthesis model, determine a phoneme sequence based on the target text, encode the phoneme sequence, and perform a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map;

[0038] a speech sequence determination module configured to determine speech data based on the phoneme feature map through the speech synthesis model, and determine a speech time sequence based on the speech data;

[0039] a mouth shape sequence determination module configured to determine a target mouth shape based on the speech time sequence through the speech synthesis model, and determine a mouth shape time sequence based on the speech time sequence and the target mouth shape;

[0040] a robot control module configured to control a motor of a mouth portion to perform corresponding movement based on the mouth shape time sequence while playing the speech data, so that the bionic robot produces a corresponding mouth shape.

[0041] In a third aspect, the present application provides an electronic device, comprising:

[0042] a memory configured to save a computer program;

[0043] a processor configured to execute the computer program to implement the bionic robot control method described above.

[0044] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program; wherein the computer program is executed by a processor to implement the bionic robot control method described above.

[0045] In the present application, the target text is obtained through the speech synthesis model, the phoneme sequence is determined based on the target text, the phoneme sequence is encoded, and the obtained encoded sequence is subjected to a preset convolution operation to obtain the corresponding phoneme feature map; the speech data is determined based on the phoneme feature map through the speech synthesis model, and the speech time sequence is determined based on the speech data; the target mouth shape is determined based on the speech time sequence through the speech synthesis model, and the mouth shape time sequence is determined based on the speech time sequence and the target mouth shape; while playing the speech data, the motor of the mouth part is controlled to move correspondingly based on the mouth shape time sequence, so that the bionic robot produces the corresponding mouth shape. As can be seen from the above, the present application obtains the target text through the speech synthesis model, determines the phoneme sequence according to the target text, encodes the phoneme sequence, and performs a preset convolution operation on the obtained encoded sequence to obtain the corresponding phoneme feature map; with the aid of the speech synthesis model, the speech data is determined according to the phoneme feature map, and the speech time sequence is determined according to the speech data; using the speech synthesis model, the target mouth shape is determined based on the speech time sequence, and the mouth shape time sequence is determined in combination with the speech time sequence and the target mouth shape; when playing the speech data, the motor of the mouth part is controlled to move according to the mouth shape time sequence, so that the bionic robot presents the corresponding mouth shape. In this way, the present application can generate speech data through the speech synthesis model while outputting the mouth shape time sequence, so as to realize accurate speech mouth shape synchronization of the bionic robot, and to a certain extent, reduce the waiting time when the robot interacts with the user. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0047] Figure 1 A bionic robot control method flow chart disclosed by the present application;

[0048] Figure 2 A specific bionic robot control method flow chart disclosed by the present application;

[0049] Figure 3 An audio and phoneme alignment schematic diagram disclosed by the present application;

[0050] Figure 4 A mouth part driving motor and control direction schematic diagram of a bionic robot disclosed by the present application;

[0051] Figure 5A schematic structural diagram of a bionic robot control device disclosed in the application;

[0052] Figure 6 A structural diagram of an electronic device disclosed in the application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the application.

[0054] Currently, for bionic robot control, there is a technical solution of performing time-frequency analysis and threshold segmentation on a voice file, but this solution can only realize the binary action of mouth opening and closing, cannot complete fine control such as subtle changes of lips, and is difficult to present a natural lip shape. Another technical solution needs to use at least three artificial intelligence models such as a large language model, a voice synthesis model, a lip shape generation model, and a voice-lip shape alignment model, which not only needs a large amount of labeled data for training and has a large implementation difficulty, but also introduces obvious time delay in actual dialogue interaction due to multi-model processing, affecting the fluency of interaction. Therefore, the application provides a bionic robot control method, device, equipment and storage medium, which can accurately control the action of the mouth of the bionic robot while the bionic robot plays voice.

[0055] Referring to Figure 1 The embodiment of the application discloses a bionic robot control method applied to a bionic robot comprising a voice synthesis model, comprising:

[0056] Step S11, obtaining a target text through the voice synthesis model, determining a phoneme sequence based on the target text, encoding the phoneme sequence, and performing a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map.

[0057] First of all, it should be noted that the voice synthesis model in this embodiment is a model that has been trained, and in the training process of the model, the only data required is text and corresponding voice data. The mel-frequency cepstral coefficient of the voice data can be obtained by existing technology, and no additional training process is required. In addition, in the process of model training, the L1 loss function is used to supervise the length of the mel-frequency cepstral coefficient and the voice data. The L1 loss function is as follows:

[0058] ;

[0059] In the formula, Mel-frequency cepstral coefficients for model predicted output, Mel-frequency cepstral coefficients for original audio, i.e., real Mel-frequency cepstral coefficients, N is the number of Mel-frequency cepstral coefficients, Fs is a sampling rate, which is the number of audio points collected in the voice data within a preset time period, T is the length of the voice data.

[0060] Then, the supervision of the output voice signal adopts KL divergence or L2 loss of data without limitation, adopts is represented. And, L1 loss function and obtained the final model training loss function. The model training loss function is as follows:

[0061] ;

[0062] In the formula, is a weight adjustment parameter, and its value range is .

[0063] In the embodiment, the bionic robot first acquires the target text to be processed through the built-in speech synthesis model, then determines the corresponding phoneme sequence based on the target text, subsequently performs encoding operation on the phoneme sequence, and finally generates the phoneme feature map through the preset convolution operation. It should be noted that the above-mentioned target text is the text information to be converted into a voice signal by the bionic robot, which can come from multiple input modes.

[0064] In a specific embodiment, the target text can be directly input by the user through the text input interface configured by the bionic robot, such as the text content transmitted through a touch screen, an external keyboard or the like. In another specific embodiment, the target text can also be obtained by the bionic robot through the speech recognition module to convert the collected voice signal. When the user interacts with the bionic robot through voice, the speech recognition module converts the voice into corresponding text information, i.e., as the target text, thereby realizing indirect input of voice to text.

[0065] In the embodiment, when determining the phoneme sequence based on the target text, different language types need to be processed. When there is Chinese in the target text, the pinyin of Chinese is determined as the first phoneme; when there is English in the target text, the phonetic symbol of English is determined as the second phoneme. At the same time, in order to ensure the integrity and continuity of the phoneme sequence, a preset blank data is also introduced, which is used to separate different language units, including between single Chinese, between words, between single Chinese and words, and the starting position and ending position of the target text, so that the phoneme sequence can accurately correspond to the overall structure of the target text.

[0066] In the encoding of the phoneme sequence, the phonemes in the phoneme sequence and the preset blank data are converted based on a preset number set. Among them, the preset blank data is uniformly converted into a zero value in the preset number set, which can clearly distinguish meaningful phonemes and blank information for separation, and facilitate subsequent model processing. After the encoding operation is completed, an encoded sequence is obtained, which will be used as the input of the subsequent convolution operation.

[0067] In this embodiment, the encoded sequence is subjected to a preset convolution operation by the embedding layer, the convolution layer and the neural network of the speech synthesis model, and finally a phoneme feature map capable of representing the overall features of the phoneme sequence is obtained.

[0068] Step S12, determining speech data based on the phoneme feature map by the speech synthesis model, and determining a speech time sequence based on the speech data.

[0069] In this embodiment, the speech synthesis model of the bionic robot determines the speech data based on the phoneme feature map obtained in step S11, and further generates a speech time sequence based on the speech data.

[0070] In determining the speech data, first, the first frame number corresponding to each phoneme in the phoneme feature map needs to be determined, and the phoneme feature map is sampled based on the first frame number to obtain a mel frequency cepstral coefficient. This coefficient can reflect the speech spectrum characteristics and effectively simulate the human ear's perception characteristics of sound. Then, the mel frequency cepstral coefficient is up-sampled to supplement the detailed information by interpolation and other methods to obtain the corresponding speech data.

[0071] In generating the speech time sequence, the first frame numbers corresponding to each phoneme in the speech data are added first to obtain the second frame number of the speech data. Then, the speech time sequence of the speech data is determined based on the second frame number, a preset sampling rate and the length of the current frame. Among them, the preset sampling rate is the number of audio points collected in the speech data within a preset time period.

[0072] Step S13, determining a target mouth shape based on the speech time sequence by the speech synthesis model, and determining a mouth shape time sequence based on the speech time sequence and the target mouth shape.

[0073] In this embodiment, the speech synthesis model of the bionic robot determines the corresponding target mouth shape based on the speech time sequence, and generates the mouth shape time sequence in combination with the speech time sequence and the target mouth shape.

[0074] In determining the target lip shape, firstly, each target phoneme in the speech time series is identified, and one target phoneme is selected as the current phoneme. The first start time and first end time of the current phoneme in the speech time series are then determined. Furthermore, determining the first start time and first end time requires identifying the end time of the previous phoneme adjacent to the current phoneme in the speech time series as the first start time of the current phoneme. The start time of the first phoneme in the speech time series is set to a preset time. Then, the first end time of the current phoneme is determined based on the duration of each current frame and the frame number of the current phoneme.

[0075] After determining the first start time and the first end time of the current phoneme, each target lip shape corresponding to the current phoneme is determined from a preset lip shape set, and one target lip shape is selected from each target lip shape as the current lip shape. Further, based on the first start time, the first end time, the number of each target lip shape, and the current lip shape, the second start time and the second end time of the current lip shape are determined.

[0076] The process proceeds to the step of selecting one target lip shape from among the target lip shapes as the current lip shape, until all target lip shapes corresponding to the current phoneme have been selected. Then, the process proceeds to the step of selecting one target phoneme from among the target phonemes as the current phoneme, until all target phonemes have been selected.

[0077] When all the target phonemes have been selected, the corresponding mouth shape time sequence is determined based on the second start time and the second end time of each target mouth shape of each target phoneme.

[0078] Step S14: While playing the voice data, control the motor of the mouth to move accordingly based on the lip-sync time sequence so that the bionic robot can produce the corresponding lip-sync.

[0079] In this embodiment, when the bionic robot plays voice data, it synchronously controls the motors of its mouth to perform corresponding movements based on the lip-sync time sequence generated in step S13, ensuring that the lip movements are consistent with the speech. Specifically, the lip-sync time sequence provides precise timing instructions for the motor movements of the mouth. The motors adjust their amplitude and speed according to these instructions, driving the mouth structure to make the corresponding lip movements. Through this synchronous control, the bionic robot presents natural and coordinated mouth movements when speaking, enhancing the realism of the interaction.

[0080] As can be seen from the above, the target text is obtained through the speech synthesis model, the phoneme sequence is determined according to the target text, the phoneme sequence is encoded, and then the preset convolution operation is performed on the obtained encoded sequence to obtain the corresponding phoneme feature map; with the help of the speech synthesis model, the speech data is determined according to the phoneme feature map, and the speech time sequence is determined according to the speech data; the speech synthesis model is used to determine the target mouth shape based on the speech time sequence, and the mouth shape time sequence is determined by combining the speech time sequence and the target mouth shape; when playing the speech data, the mouth motor is controlled according to the mouth shape time sequence, so that the bionic robot presents the corresponding mouth shape. In this way, the speech synthesis model can generate speech data while outputting the mouth shape time sequence, so that the bionic robot realizes accurate speech mouth shape synchronization, and the waiting time of the robot when interacting with the user is reduced to a certain extent.

[0081] The technical solutions of the embodiments of the present application will be specifically described below in conjunction with the schematic diagram shown in the accompanying drawings. Figure 2

[0082] Specifically, the text (indicated by , that is, the target text) that needs to be synthesized by speech is converted into phonemes. When the text to be synthesized is Chinese, the corresponding Chinese pinyin is used as the phoneme, and when the text to be synthesized is English or other languages, the corresponding phonetic symbols are used as the phonemes. In order to unify the phonemes of various languages, both pinyin and phonetic symbols are transcribed using the International Phonetic Alphabet. In addition, in order to make the generated speech data have the pauses when speaking like a person when playing, a blank data is added before and after each Chinese character or word to obtain the phoneme time sequence (that is, the phoneme sequence) . Wherein, represents the i-th phoneme. For example, when the text to be speech broadcast is , the phoneme time sequence is , wherein, represents a blank data.

[0083] Next, all phonemes are uniquely encoded, so that the values in the phoneme time sequence are converted into numbers, wherein, is represented by a special number 0. The phoneme information is embedded through an embedding layer and a convolution layer, and is encoded into a feature map (that is, a phoneme feature map) through a neural network. The feature map obtains the length of each phoneme through a prediction module. Its dimension can be represented as (B, N), where B is BatchSize, representing the number of input texts, and N is the length of the phoneme sequence, that is, the length of , the value of which represents the number of speech frames of the corresponding phoneme. In addition, when inferring the speech synthesis, B is often taken as 1, for the sake of simplifying the subsequent representation, B=1 is taken, which is degraded from a two-dimensional matrix to a one-dimensional vector form, and is represented by .

[0084] Further, by accumulating The length of the time frame of the predicted speech (i.e., the speech data) (i.e., the speech time sequence) can be calculated according to the formula and the mel frequency cepstral coefficient is obtained by sampling the phoneme feature map. Then, the generated speech data is obtained through the up-sampling network. The length of the speech data The length of the time frame, the sampling rate SR, and the length of each frame F are jointly determined, and the length of the speech data, the sampling rate, the length of the time frame, and the length of each frame can be represented by the following formula:

[0085] ;

[0086] Through the above process, the speech data and the corresponding speech time sequence can be obtained, and the start time and the end time of each phoneme can be corresponded on the waveform of the speech data. The input text and the output speech data and the phoneme alignment effect can be as shown in Figure 3 . Specifically, the input text is "hello world, you good world", and the generated speech data is as shown in Figure 3 the waveform data drawn, the text in the yellow box below represents the text corresponding to the audio pronunciation at the current time, and the text above represents the phoneme corresponding to the audio pronunciation at the current time. When the length of the blank phoneme "silence" is less than two time frame lengths, the blank phoneme is ignored and merged into the previous valid phoneme. The time corresponding to the dashed line before and after each phoneme is the start time and the end time of the phoneme.

[0087] Then, in order to visually represent the mouth shape on the robot, the pronunciation process of the phoneme is summarized, and a static picture is used to represent the position state of the mouth opening and the lip shape when pronouncing. A preset M visual mouth shape image represents a single visual mouth shape, each image represents a static pronunciation process, and an ID is used for representation. In this embodiment, M is 12, and the visual mouth shape picture is as shown in Table 1. For example, ID=0 represents the state of teeth occlusion and naturally closed lips.

[0088] Table 1

[0089]

[0090] ​​Since the visualized mouth shape is a static state, but the pronunciation of part of the phonemes is a changing process, such as the representation of "ao", the mouth is round, and the process of changing from large to small. Therefore, each phoneme can be combined with multiple visualized mouth shapes to form a continuous change. When the visualized mouth shape ID of an phoneme is only 1, the visualized mouth shape is statically represented at the start time to the end time of the current phoneme, and when the visualized mouth shape ID of an phoneme has multiple, the time length is evenly distributed from the start time to the end time of the current phoneme, and each visualized mouth shape is dynamically displayed. The calculation formula of the start time of each visualized mouth shape is as follows:

[0091] ;

[0092] In the formula, The start time of the jth visualized mouth shape of the ith phoneme is represented, and M represents the number of visualized mouth shapes of the ith phoneme. Similarly, the end time of each visualized mouth shape is represented by the following expression.

[0093] ;

[0094] In the formula, The end time of the jth visualized mouth shape of the ith phoneme is represented.

[0095] In addition, And The calculation formula is as follows:

[0096] ;

[0097] ;

[0098] From the above formula, the start time of each phoneme time is the end time of the previous phoneme, and the start time of the first phoneme is 0. Where F represents the time length of each frame, and F can be set to 0.032 seconds.

[0099] In addition, the correspondence between part of the phonemes and the visualized mouth shape ID can refer to Table 2 as follows.

[0100] Table 2

[0101]

[0102] In order to display the visualized mouth shape, when designing the robot bionic head, the movement of the mouth is controlled by 8 motors, and the motor control position of each visualized mouth shape is obtained by calibration. When the output visualized mouth shape needs to be controlled, the motor is controlled to the corresponding position, and the mouth motor control direction can refer to Figure 4As shown in the figure. Control points ①, ②, ③ are used to control the movement of the upper lip, control points ④, ⑤, ⑥ are used to control the movement of the lower lip, so that the lips can have a change from ID=1 to ID=2, control point ⑦ is used to control the movement of the lower jaw, so that the mouth has the state of overall opening and closing, control point ⑧ is used to control the stretching of the tongue, so that the tongue can change from ID=6 to ID=9.

[0103] According to the above process, the time sequence of visualized mouth shape ID corresponding to the speech data (i.e. the mouth shape time sequence) can be obtained, and when playing the speech data, the mouth motor is driven according to the corresponding visualized mouth shape ID time sequence to realize the visualized output of the mouth shape. For normal speed speech, the duration of a single phoneme is not more than 100ms, a single phoneme can be divided into 1-3 visualized mouth shape actions, and then the control frequency of 10-30Hz can be realized to drive the motor to visualize the mouth shape, and the visualized mouth shape has M kinds, which is more precise and natural than the binary control of only controlling the opening and closing of the mouth. Further, by adopting fixed frequency control precision, the position of the mouth shape motor is controlled by interpolation, so that a more smooth and natural artificial mouth shape can be obtained.

[0104] In addition, when the text to be reported by voice is long, the processing time of speech synthesis and generation of mouth shape time sequence is also long. In order to reduce the delay waiting time, the text paragraph for speech synthesis is divided into different sentences by punctuation, and the sentences are processed by segmentation to obtain different sentences. After completing the speech synthesis and the generation of the mouth shape time sequence of the current sentence, the playing task is issued to the robot to start the speech data playing and the corresponding mouth shape driving, and at the same time, the speech synthesis and the generation of the mouth shape time sequence of the next sentence are performed.

[0105] Correspondingly, referring to Figure 5 As shown in the figure, the embodiment of the present application provides a bionic robot control device applied to a bionic robot comprising a speech synthesis model, comprising:

[0106] The feature map determination module 11 is configured to acquire a target text through the speech synthesis model, determine a phoneme sequence based on the target text, encode the phoneme sequence, and perform a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map;

[0107] The speech sequence determination module 12 is configured to determine speech data based on the phoneme feature map through the speech synthesis model, and determine a speech time sequence based on the speech data;

[0108] The mouth shape sequence determination module 13 is configured to determine a target mouth shape based on the speech time sequence through the speech synthesis model, and determine a mouth shape time sequence based on the speech time sequence and the target mouth shape;

[0109] The robot control module 14 is configured to control the motor of the mouth to move according to the mouth shape time sequence while playing the voice data, so that the bionic robot generates the corresponding mouth shape.

[0110] As can be seen from the above, the target text is obtained through the speech synthesis model, the phoneme sequence is determined according to the target text, the preset convolution operation is performed on the obtained coded sequence, and the corresponding phoneme feature map is obtained; the speech data is determined according to the phoneme feature map by means of the speech synthesis model, and the speech time sequence is determined according to the speech data; the target mouth shape is determined based on the speech time sequence by means of the speech synthesis model, and the mouth shape time sequence is determined according to the speech time sequence and the target mouth shape; when playing the voice data, the mouth motor is controlled to move according to the mouth shape time sequence, so that the bionic robot presents the corresponding mouth shape. In this way, the speech synthesis model can be used to generate voice data while outputting the mouth shape time sequence, so that the bionic robot realizes accurate voice and mouth shape synchronization, and the waiting time of the robot and the user during interaction is reduced to a certain extent.

[0111] In some embodiments, the feature map determination module 11 specifically includes:

[0112] The first phoneme determination unit is configured to determine the pinyin of the Chinese as the first phoneme when the target text contains Chinese;

[0113] The second phoneme determination unit is configured to determine the phonetic symbol of the English as the second phoneme when the target text contains English;

[0114] The phoneme sequence determination unit is configured to determine the phoneme sequence based on the first phoneme, the second phoneme, and the preset blank data; the preset blank data is data between single Chinese in the target text, between words, and between the single Chinese and the words, and data of the start position and the end position of the target text.

[0115] In some embodiments, the feature map determination module 11 specifically includes:

[0116] The phoneme encoding unit is configured to convert the phonemes in the phoneme sequence and the preset blank data based on a preset number set to complete the encoding operation of the phoneme sequence, and obtain a corresponding coded sequence; wherein the preset blank data is converted to zero in the encoding operation.

[0117] The sequence convolution unit is configured to perform a preset convolution operation on the coded sequence through the embedding layer, the convolution layer, and the neural network of the speech synthesis model, and obtain a corresponding phoneme feature map.

[0118] In some embodiments, the speech sequence determining module 12 specifically comprises:

[0119] a frame number determining unit configured to determine a first frame number corresponding to each phoneme in the phoneme feature map;

[0120] a coefficient determining unit configured to sample the phoneme feature map based on the first frame number to obtain a mel-frequency cepstral coefficient;

[0121] a coefficient sampling unit configured to up-sample the mel-frequency cepstral coefficient to obtain corresponding speech data.

[0122] In some embodiments, the speech sequence determining module 12 specifically comprises:

[0123] a frame number adding unit configured to add the first frame number corresponding to each phoneme in the speech data to obtain a second frame number of the speech data;

[0124] a speech sequence determining unit configured to determine a speech time sequence of the speech data based on the second frame number, a preset sampling rate and a current time length of each frame; the preset sampling rate is a number of audio points collected in the speech data within a preset time period.

[0125] In some embodiments, the mouth shape sequence determining module 13 specifically comprises:

[0126] a third phoneme determining unit configured to determine each target phoneme in the speech time sequence;

[0127] a first time point determining unit configured to select one target phoneme from the target phonemes as a current phoneme, and determine a first start time point and a first end time point of the current phoneme in the speech time sequence;

[0128] a mouth shape determining unit configured to determine each target mouth shape corresponding to the current phoneme from a preset mouth shape set, and select one target mouth shape from the target mouth shapes as a current mouth shape;

[0129] a second time point determining unit configured to determine a second start time point and a second end time point of the current mouth shape based on the first start time point, the first end time point, a number of the target mouth shapes and the current mouth shape;

[0130] a first step jumping unit configured to jump to a step of selecting one target mouth shape from the target mouth shapes as a current mouth shape until each target mouth shape corresponding to the current phoneme is selected completely;

[0131] The second step jumping unit is configured to jump to the step of selecting a target phoneme from the target phonemes as a current phoneme until all the target phonemes are selected.

[0132] The mouth shape sequence determining unit is configured to determine a corresponding mouth shape time sequence based on the second start time and the second end time of the target mouth shape of each target phoneme when all the target phonemes are selected.

[0133] In some embodiments, the first time determining unit specifically comprises:

[0134] The start time determining sub-unit is configured to determine, in the speech time sequence, an end time of a previous phoneme adjacent to the current phoneme as a first start time of the current phoneme; and the start time of a first phoneme in the speech time sequence is set as a preset time.

[0135] The end time determining sub-unit is configured to determine, based on a time length of each frame and a frame number of the current phoneme, a first end time of the current phoneme.

[0136] Further, the embodiment of the present application further discloses an electronic device, Figure 6 is an electronic device 20 structure diagram shown according to an exemplary embodiment, the contents in the figure cannot be considered as any limitation on the use range of the present application. The electronic device 20, specifically can include: at least one processor 21, at least one memory 22, power supply 23, communication interface 24, input output interface 25 and communication bus 26. Wherein, the memory 22 is used for storing computer program, the computer program is loaded and executed by the processor 21, to realize the related steps in the bionic robot control method disclosed in any preceding embodiment. In addition, the electronic device 20 in the embodiment specifically can be electronic computer.

[0137] In the embodiment, the power supply 23 is used for providing working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input output interface 25 is used for obtaining external input data or outputting data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0138] In addition, the memory 22 as the carrier of resource storage can be read-only memory, random access memory, disk or optical disk, etc., and the resources stored thereon can include operating system 221, computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0139] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs capable of performing other specific work in addition to the computer programs capable of performing the bionic robot control method disclosed in any of the foregoing embodiments and executed by the electronic device 20.

[0140] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the bionic robot control method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0141] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0142] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in general in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0143] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0144] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be intended to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the stated element.

[0145] The above detailed description of the technical solutions provided by the present application has been provided, and the principles and implementation manners of the present application have been described by applying specific examples. The above description of the examples is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description of the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method of controlling a biomimetic robot, characterized by, The application is applied to a bionic robot comprising a speech synthesis model, and includes: acquiring target text through the speech synthesis model, determining a phoneme sequence based on the target text, encoding the phoneme sequence, and performing a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map; determining speech data based on the phoneme feature map through the speech synthesis model, and determining a speech time sequence based on the speech data; determining a target mouth shape based on the speech time sequence through the speech synthesis model, and determining a mouth shape time sequence based on the speech time sequence and the target mouth shape; controlling the motor of the mouth to move correspondingly based on the mouth shape time sequence while playing the speech data, so that the bionic robot produces a corresponding mouth shape.

2. The biomimetic robot control method of claim 1, wherein, The method for determining a phoneme sequence based on the target text includes: when there is Chinese in the target text, determining the pinyin of the Chinese as a first phoneme; when there is English in the target text, determining the phonetic symbol of the English as a second phoneme; determining a phoneme sequence based on the first phoneme, the second phoneme, and preset blank data; the preset blank data is data between individual Chinese characters, between words, and between individual Chinese characters and words in the target text, as well as data of the start position and end position of the target text.

3. The biomimetic robot control method of claim 2, wherein, The method for encoding the phoneme sequence and performing a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map includes: transforming the phonemes in the phoneme sequence and the preset blank data based on a preset number set to complete the encoding operation of the phoneme sequence to obtain a corresponding encoded sequence; wherein the preset blank data is transformed into a zero value in the preset number set in the encoding operation; performing a preset convolution operation on the encoded sequence through the embedding layer, the convolution layer, and the neural network of the speech synthesis model to obtain a corresponding phoneme feature map.

4. The biomimetic robot control method of claim 1, wherein, The method for determining speech data based on the phoneme feature map includes: determining a first frame number corresponding to each phoneme in the phoneme feature map; sampling the phoneme feature map based on the first frame number to obtain a mel-frequency cepstral coefficient; upsampling the mel-frequency cepstral coefficient to obtain corresponding speech data.

5. The biomimetic robot control method of claim 4, wherein, The method for determining a speech time sequence based on the speech data includes: adding the first frame number corresponding to each phoneme in the speech data to obtain a second frame number of the speech data; determining a speech time sequence of the speech data based on the second frame number, a preset sampling rate, and the length of each frame at present; the preset sampling rate is the number of audio points collected in the speech data within a preset time period.

6. The method of controlling a biomimetic robot according to any one of claims 1 to 5, wherein, The method for determining a target mouth shape based on the speech time sequence, and determining a mouth shape time sequence based on the speech time sequence and the target mouth shape includes: determining each target phoneme in the speech time sequence; selecting a target phoneme from each target phoneme as a current phoneme, and determining a first start time and a first end time of the current phoneme in the speech time sequence; determine each target articulator corresponding to the current phoneme from a preset articulator set, and select one target articulator from each target articulator as a current articulator; determine a second start time and a second end time of the current articulator based on the first start time, the first end time, the number of each target articulator, and the current articulator; jump to the step of selecting one target articulator from each target articulator as a current articulator until each target articulator corresponding to the current phoneme is selected; jump to the step of selecting one target phoneme from each target phoneme as a current phoneme until each target phoneme is selected; when each target phoneme is selected, determine the corresponding articulator time sequence based on the second start time and the second end time of each target articulator of each target phoneme.

7. The biomimetic robot control method of claim 6, wherein, The determination of the first start time and the first end time of the current phoneme in the speech time sequence comprises: In the speech time sequence, the end time of the previous phoneme adjacent to the current phoneme is determined as the first start time of the current phoneme; wherein the start time of the first phoneme in the speech time sequence is set as a preset time; determine the first end time of the current phoneme based on the duration of each current frame and the number of frames of the current phoneme.

8. A biomimetic robot control device, characterized by, Applied to a bionic robot comprising a speech synthesis model, comprising: a feature map determination module for obtaining a target text through the speech synthesis model, determining a phoneme sequence based on the target text, encoding the phoneme sequence, and performing a preset convolution operation on the obtained encoded sequence to obtain a corresponding phoneme feature map; a speech sequence determination module for determining speech data based on the phoneme feature map through the speech synthesis model, and determining a speech time sequence based on the speech data; an articulator sequence determination module for determining a target articulator based on the speech time sequence through the speech synthesis model, and determining an articulator time sequence based on the speech time sequence and the target articulator; a robot control module for controlling the motor of the mouth to move correspondingly based on the articulator time sequence while playing the speech data, so that the bionic robot produces corresponding articulators.

9. An electronic device, comprising: comprising: a memory for saving a computer program; a processor for executing the computer program to implement the bionic robot control method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, for saving a computer program; wherein the computer program is executed by a processor to implement the bionic robot control method of any one of claims 1 to 7.

Images