Digital human driving method, device and system based on speech recognition

Abstract

The present invention relates to the field of speech recognition technology, and in particular to a method, device and system for driving a digital human based on speech recognition, the method comprising: obtaining audio data of each sound source in a target area; determining a target independent component voice belonging to a voice of an interactive user; determining a similar voice group; determining a distance difference between a first reference position of an interactive user to a first position and a second reference position of an interactive user to a second position according to a first acquisition moment of a first target independent component voice, a second acquisition moment of a second target independent component voice and a sound propagation speed; determining a target position of the interactive user according to a distance and a distance difference between the first reference position and the second reference position; and driving a digital human to face the interactive user using the target position of the interactive user in a target area. In this way, the present invention improves the interactive flexibility between a digital human and an audience, ensures the audience's sense of participation in the interactive process, and improves the audience's interactive experience.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a digital human driving method, device and system based on speech recognition. Background Art

[0002] Digital human is a digital human image close to human image created by digital technology. Digital human drive refers to the use of digital technology or algorithm-driven methods to simulate or control the behavior and interaction of digital human. By applying digital human technology in exhibition halls or exhibitions, voice recognition technology can be used to identify the audience's questions or interactive sentences, and the corresponding answer sentences can be output to provide intelligent guidance for the audience and realize voice interaction with the audience. When the audience interacts with the digital human by voice, the digital human will also have a motion control module to grasp the digital human's movements and expressions. In the process of interaction with the audience, the digital human combines movements and expressions to provide the audience with more intelligent and interactive exhibition accompaniment and information services.

[0003] In some scenarios, when the digital human and the audience are interacting by voice, the digital human will also make movements and expressions to make the interaction between the digital human and the audience more vivid and rich. When the digital human makes movements and expressions, the face of the digital human always faces a fixed direction. No matter where the audience interacting with it is, the direction of the digital human's face remains unchanged. In this way, the interaction flexibility between the digital human and the audience is poor, and the audience's sense of participation in the interaction process is low, resulting in a poor interactive experience for the audience. Summary of the invention

[0004] In order to solve the technical problem that the interaction flexibility between digital humans and audiences is poor, resulting in poor interactive experience for audiences, the purpose of the present invention is to provide a digital human driving method, device and system based on speech recognition. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a digital human driving method based on speech recognition, comprising: acquiring audio data of each sound source in a target area through sound collection devices located at different positions in the target area; determining a target independent component voice belonging to the voice of an interactive user from the audio data of each sound source; determining a similar voice group, the similar voice group including a first target independent component voice of a sound collection device located at a first position and a second target independent component voice of a sound collection device located at a second position; determining a distance difference between a first reference position of the interactive user to the first position and a second reference position of the interactive user to the second position according to a first collection moment of the first target independent component voice, a second collection moment of the second target independent component voice and a sound propagation speed; determining a target position of the interactive user in the target area according to the distance between the first reference position and the second reference position and the distance difference; and driving the digital human to face the interactive user using the target position of the interactive user in the target area.

[0006] Optionally, determining the target independent component speech belonging to the speech of the interactive user from the audio data of each sound source includes: performing independent component analysis on the audio data collected by the sound collection devices at different locations to obtain the sound frequency sequence of the independent component audio of each sound source at different times; determining the validity of the independent component audio according to the first number of extreme points in the sound frequency sequence, the first time interval between adjacent maximum points, the second time interval between adjacent minimum points, and the variance of the sound frequency in the sound frequency sequence; determining the independent component speech belonging to the speech based on the validity of the independent component audio; performing speech recognition on the independent component speech to obtain text information of the independent component speech; segmenting the text information to obtain multiple words; determining the interactive user who interacts with the digital human through the correlation between the words and the exhibition data corresponding to the digital human; and determining the independent component speech of the interactive user as the target independent component speech.

[0007] Optionally, determining the validity of the independent component audio based on a first number of extreme points in the sound frequency sequence, a first time interval between adjacent maximum points, a second time interval between adjacent minimum points, and a variance of the sound frequency in the sound frequency sequence includes: determining a first average value of the first time interval between all adjacent maximum points in the sound frequency sequence, a second average value of the second time interval between all adjacent minimum points in the sound frequency sequence; calculating a first sum of the first average value and the second average value, and calculating a first product between the first number and the variance; determining a first ratio between the first sum and the first product, and performing inverse proportional normalization processing on the first ratio to obtain the validity of the independent component audio.

[0008] Optionally, the sound collection device includes a sound collection device at a first position in the target area and a sound collection device at a second position in the target area, and determining the similar speech group includes: determining the similar speech group according to the similarity between the target independent component speech of the sound collection device at the first position in the target area and the sound collection device at the second position in the target area.

[0009] Optionally, based on the similarity between the target independent component speech of the sound collection device at the first position of the target area and the target independent component speech of the sound collection device at the second position of the target area, determining the similar speech group includes: determining a first average value of a first time interval between all adjacent maximum points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position, and a second average value of a second time interval between all adjacent minimum points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position; determining a third average value of a third time interval between all adjacent maximum points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position, and a fourth average value of a fourth time interval between all adjacent minimum points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position; determining the first average value of each time interval The first difference between the sound frequency of the target independent component speech of the sound collection device at the first position and the sound frequency of the target independent component speech of the sound collection device at the second position is calculated, and the first difference values ​​are superimposed to obtain a superposition value; a second sum of the first average value and the second average value, a third sum of the third average value and the fourth average value, and a second ratio of the second sum value to the third sum value are calculated; the second product of the absolute value of the second difference between the second ratio and the preset value and the superposition value is calculated; the second product is inversely normalized to obtain the similarity between the target independent component speech of the sound collection device at the first position and the target independent component speech of the sound collection device at the second position; the first target independent component speech of the sound collection device at the first position and the second target independent component speech of the sound collection device at the second position with the highest similarity are determined as a similar speech group.

[0010] Optionally, determining the distance difference between the first reference position of the interactive user to the first position and the second reference position of the interactive user to the second position based on the first collection moment of the first target independent component speech, the second collection moment of the second target independent component speech and the sound propagation speed includes: determining the first moment in the sound frequency sequence of the first target independent component speech when the sound frequency is greater than a predetermined value as the first collection moment; determining the average value of the start moment of all sound collection devices at the second position starting to collect the second target independent component speech as the second collection moment; calculating the difference between the first collection moment and the second collection moment to obtain the collection duration; and determining the fourth product of the sound propagation speed and the collection duration as the distance difference.

[0011] Optionally, determining the target position of the interactive user in the target area based on the distance between the first reference position and the second reference position and the distance difference includes: determining the length of the line between the first reference position and the second reference position as the distance; calculating a fourth sum of half the distance and the distance difference; taking the first reference position as the starting point and taking a target line segment on the line connecting the first reference position and the second reference position with a length equal to the fourth sum; and determining the end point of the target line segment as the target position.

[0012] Optionally, using the target position of the interactive user in the target area to drive the digital human to face the interactive user includes: taking the vertex of the first position of the target area as the pole, and the line between the vertex and the first reference position as the polar axis; determining the angle between the line between the pole and the target position and the polar axis as the polar angle of the interactive user; when there are multiple interactive users, driving the digital human to face the interactive users at the corresponding target positions in order of polar angles from small to large; when there is only one interactive user, driving the digital human to face the interactive user at the target position.

[0013] In a second aspect, an embodiment of the present invention provides a digital human driving system based on speech recognition, comprising: an acquisition module, used to acquire audio data of each sound source in a target area through sound collection devices located at different positions in the target area; a determination module, used to determine a target independent component voice belonging to the voice of an interactive user from the audio data of each sound source; the determination module, further used to determine a similar voice group, the similar voice group including a first target independent component voice of a sound collection device located at a first position and a second target independent component voice of a sound collection device located at a second position; the determination module, further used to determine a distance difference between a first reference position of the interactive user to the first position and a second reference position of the interactive user to the second position according to a first collection moment of the first target independent component voice, a second collection moment of the second target independent component voice and a sound propagation speed; the determination module, further used to determine a target position of the interactive user in the target area according to the distance between the first reference position and the second reference position and the distance difference; and a driving module, used to drive the digital human to face the interactive user using the target position of the interactive user in the target area.

[0014] In the third aspect, an embodiment of the present invention provides a digital human driving device based on voice recognition, which includes: a processor and a memory; wherein the memory is used to store computer programs that can be run on the processor; the processor is used to execute the program stored in the memory to implement the steps of the digital human driving method based on voice recognition mentioned in the first aspect.

[0015] The present invention has the following beneficial effects: firstly, the audio data of each sound source in the target area is acquired by the sound collection devices located at different positions in the target area, then the target independent component voice belonging to the voice of the interactive user is determined from the audio data of each sound source, and then a similar voice group is determined, the similar voice group includes the first target independent component voice of the sound collection device at the first position and the second target independent component voice of the sound collection device at the second position; secondly, the distance difference between the first reference position of the interactive user to the first position and the second reference position of the interactive user to the second position is determined according to the first collection time of the first target independent component voice, the second collection time of the second target independent component voice and the sound propagation speed; then, the target position of the interactive user in the target area is determined according to the distance between the first reference position and the second reference position and the distance difference; finally, the target position of the interactive user in the target area is used to drive the digital human to face the interactive user.

[0016] In this way, the embodiment of the present invention can analyze the target independent component voice of the voice of the interactive user according to the audio data collected by the sound collection device, and then determine the similar voice group according to the target independent component voice, and use the target independent component voice of the sound collection device at different positions in the similar voice group and the reference position of each position to determine the target position of the interactive user interacting with the digital human. That is, the interactive user is positioned, and after the interactive user is positioned, the digital human is driven to face the target position where the interactive user is located. In this way, the orientation of the digital human can be flexibly adjusted according to the position of the interactive user participating in the interaction, which improves the flexibility of the interaction between the digital human and the audience, ensures the audience's sense of participation in the interactive process, and improves the audience's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a digital human driving method based on speech recognition provided by one embodiment of the present invention;

[0019] Figure 2 A schematic top view of an exhibition hall provided by an embodiment of the present invention;

[0020] Figure 3 A schematic top view of another exhibition hall provided by an embodiment of the present invention;

[0021] Figure 4 A schematic side view of a right wall provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a digital human driving system based on speech recognition provided by one embodiment of the present invention;

[0023] Figure 6 A schematic structural diagram of a digital human driving device based on speech recognition provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a digital human driving method, device and system based on voice recognition proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0025] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0026] The specific scheme of a digital human driving method, device and system based on speech recognition provided by the present invention is described in detail below with reference to the accompanying drawings.

[0027] Embodiment 1:

[0028] See also Figure 1 , which shows a flow chart of a digital human driving method based on speech recognition provided by an embodiment of the present invention, including:

[0029] S101, acquiring audio data of each sound source in the target area through sound collection devices located at different positions in the target area.

[0030] Specifically, the target area may be an exhibition hall. The sound collection device may be a microphone, and different numbers of sound collection devices may be installed for different types of target areas. Taking the exhibition hall as an example of the target area, the exhibition hall environment is usually more orderly and not too noisy, and the number of microphones does not need to be set too much, thereby improving the efficiency and accuracy of subsequent voice processing. In the embodiment of the present invention, a microphone array is preset. Taking the number of microphones as 8 as an example, they are respectively installed at the four corners of the left wall and the four corners of the right wall of the exhibition hall, wherein one is installed at each of the four corners of the left wall, and one is installed at each of the four corners of the right wall. In the embodiment of the present invention, the microphone on the left wall is recorded as a sound collection device in the first position, and the microphone on the right wall is recorded as a sound collection device in the second position. Figure 2 As shown, Figure 2 A schematic top view of an exhibition hall provided by an embodiment of the present invention, Figure 2 In the exhibition hall, a screen and a microphone are included. A digital person is displayed on the screen. The digital person is displayed at a predetermined position on the screen, and the user can interact with the digital person on the screen by voice. It is worth noting that the position and number of sound collection devices installed in the target area can be adjusted according to the actual situation. For example, in the actual situation, if the sound source is more complex or the environment is more noisy, more sound collection devices can be used and installed more densely to obtain a better separation effect when performing independent component analysis.

[0031] Furthermore, the sound source includes but is not limited to the user, the component that makes noise, etc. The audio data includes but is not limited to the voice of the interactive user who interacts with the digital human, the voice of other users who do not interact with the digital human, and various noises, etc. Taking the exhibition hall as the target area, the present invention collects the audio in the exhibition hall through all the microphones in the exhibition hall. The collection frequency can be set to 44.1kHz, and the sound waveform corresponding to each microphone is obtained. Each sampling point on the sound waveform represents the sound frequency corresponding to each moment. The collection starts when the sound frequency is greater than 100HZ, and the collection ends when there is no sound greater than 100HZ within 1.5 seconds. The maximum length of the collected voice is no more than 30 seconds. Among them, the reason for starting to collect when the sound frequency is greater than 100HZ is that the reference point of the sound frequency of human speech is usually 100HZ. Therefore, starting to collect when the sound frequency is greater than 100HZ can avoid collecting too much noise. The reason why the collection time does not exceed 30 seconds is that in actual situations, the voice duration of the interaction between the users in the exhibition hall and the digital human will not be too long. Setting the collection time to no more than 30 seconds can avoid other noises greater than 100HZ that cause continuous voice collection and avoid collecting too much noise.

[0032] S102, determining a target independent component speech belonging to the speech of the interactive user from the audio data of each sound source.

[0033] Specifically, after the audio data in the target area is collected by the above embodiment of the present invention, it is composed of multiple voices, including not only the user's voice, but also other noises. Therefore, it is necessary to separate the audio data into voice data of multiple sound sources, and then perform voice recognition on the voice data of each sound source. Analyze the correlation between the voice content and the exhibition content displayed by the digital human, and distinguish the voice participating in the digital human interaction and other non-interactive voices through the correlation.

[0034] Furthermore, when determining the target independent component speech of the speech belonging to the interactive user, as an optional embodiment of the present invention, firstly, independent component analysis is performed on the audio data collected by the sound collection devices at different positions to obtain the sound frequency sequence of the independent component audio of each sound source at different times; then, the validity of the independent component audio is determined according to the first number of extreme points in the sound frequency sequence, the first time interval between adjacent maximum points, the second time interval between adjacent minimum points and the variance of the sound frequency in the sound frequency sequence; then, based on the validity of the independent component audio, the independent component speech belonging to the speech is determined; secondly, speech recognition is performed on the independent component speech to obtain text information of the independent component speech; then, the text information is segmented to obtain multiple words; and the interactive user interacting with the digital human is determined through the correlation between the words and the exhibition data corresponding to the digital human; finally, the independent component speech of the interactive user is determined as the target independent component speech.

[0035] Specifically, when the embodiment of the present invention performs independent component analysis on the audio data collected by the sound collection devices at different positions, the audio data collected by the sound collection devices are first preprocessed. The preprocessing process includes establishing a sampling matrix and centralizing the data of the sampling matrix and whitening the data. In the embodiment of the present invention, the process of establishing the sampling matrix is ​​explained by taking the n microphones at the first position in the exhibition hall as an example: at the i-th moment, the sound frequency vector collected by the microphone is , the sound frequencies of n microphones at m moments form a sampling matrix of n rows and m columns , is the element in the sampling matrix, that is, the sound frequency. If there are n sound sources, the independent audio data of the jth sound source is recorded as , then n sound sources form a signal source matrix with n rows and m columns , is the element in the signal source matrix, i.e., the sound source. It can be considered as a linear combination of n sound sources at the i-th moment, that is , then for the sampling matrix With signal source matrix ,satisfy: .in, to for The elements in is the coefficient matrix, Each element in represents the corresponding coefficient of each linear combination.

[0036] Furthermore, after establishing the sampling matrix of the first position, the sampling matrix is ​​first centralized, that is, the mean is removed, so that the mean of all elements in the sampling matrix is ​​zero. The sampling matrix after data centralization is whitened, that is, the correlation between variables is removed, and the whitened matrix is ​​obtained. . Then use the non-Gaussian maximization principle (Fast Independent Component Analysis, FICA) to randomly initialize a weight matrix , by making The non-Gaussian maximum pair weight matrix Iterate and finally obtain .in, represents the independent component matrix after unmixing, represents the weight matrix of the final iteration, Represents the whitening matrix. Each row of the independent component matrix represents the independent component audio of a sound source, and the independent component matrix of the microphone at the first position is obtained. Then, the sound frequency sequence of the independent component audio at different times in the independent component matrix is ​​constructed, and the sound frequency sequence is constructed with the horizontal axis as time and the vertical axis as sound frequency. Similarly, the independent component matrices of microphones at other positions and the corresponding sound frequency sequences are obtained.

[0037] Furthermore, since the obtained independent component audio may not all be valid user voices, but may also contain noise, the effectiveness of each independent component audio is analyzed and screened to exclude noise components. This can reduce the amount of calculation, improve the efficiency of subsequent processing, avoid misjudgment of subsequent analysis caused by noise, and improve the accuracy of analysis. The effectiveness can reflect the possibility that the independent component audio is the user's voice. The greater the effectiveness, the more likely it is the user's voice, and the smaller the effectiveness, the more likely it is other noise.

[0038] Furthermore, the first number of extreme points refers to the sum of the number of maxima and the number of minima. When determining the validity of the independent component audio, as an optional embodiment of the present invention, first determine the first average value of the first time interval between all adjacent maximum points of the sound frequency sequence, and the second average value of the second time interval between all adjacent minimum points of the sound frequency sequence; then calculate the first sum of the first average value and the second average value, and calculate the first product between the first number and the variance; finally determine the first ratio between the first sum and the first product, perform inverse proportional normalization on the first ratio, and obtain the validity of the independent component audio.

[0039] Specifically, the validity of the independent component audio in the embodiment of the present invention can be calculated using the following formula:

[0040]

[0041] In the above formula, Indicates the first position The validity of the tth independent component audio of the sound collection device. Indicates the first position The first average value of the first time interval of all adjacent maximum points in the sound frequency sequence of the t-th independent component audio of the sound collection device. Indicates the first position A second average value of a second time interval between all adjacent minimum value points in the sound frequency sequence of the t-th independent component audio of the sound collecting device. Indicates the first position The first number of extreme value points in the sound frequency sequence of the t-th independent component audio of the sound collecting device. Indicates the first position The variance of the sound frequency in the sound frequency sequence of the t-th independent component audio of the sound collection device. Represents the inverse normalization function, which is used to Perform inverse proportional normalization.

[0042] It is worth noting that The smaller the number, the higher the position. The smaller the time interval between adjacent maximum and minimum points of the t-th independent component audio of the sound collection device, the smaller the time interval between extreme points, which means that the extreme points are more compact and continuous. As for noise, it is usually irregular and discontinuous, so the The smaller the value, the more likely it is to be in the first position. The more likely the t-th independent component audio of the sound collection device is human voice, the greater the validity. Similarly, the validity of all independent component audio collected by the sound collection devices at all locations is obtained.

[0043] Furthermore, when determining the independent component speech belonging to speech based on the validity of the independent component audio, a validity threshold can be preset, and the independent component audio whose validity is less than the validity threshold is considered to be noise, and all independent component audio whose validity is greater than or equal to the validity threshold is recorded as independent component speech. The validity threshold can be determined according to actual conditions, and in the embodiment of the present invention, the value is 0.3.

[0044] Furthermore, in the embodiment of the present invention, the content displayed by the digital human is determined. In order to determine which users are interacting with the digital human, the embodiment of the present invention determines the interactive users interacting with the digital human by comparing the correlation between the independent component voice of the user and the preset matching word library of the content displayed by the digital human. Among them, the preset matching word library contains exhibition data such as key words related to the exhibition content to be displayed by the digital human and the key words that the user may mention. Among them, the exhibition data mainly includes three categories, namely, exhibition content category, related content category and audience emotion category. The exhibition content category is used to display the exhibition theme, such as technology, digital human and all content vocabulary involved in the exhibition content; the related content category is based on the related content involved in the exhibition theme, such as history, background and meaning; the audience emotion category is used to display the audience's emotions, such as magic, shock and unforgettable. Further, weights can be assigned to different types of exhibition data to improve the accuracy of matching. For example, the weight of the exhibition content category is set to 3, the weight of the related content category is set to 2, and the weight of the audience emotion category is set to 1.

[0045] Furthermore, when performing speech recognition on the independent component speech, a speech-to-text tool is used to perform speech recognition on all independent component speech to obtain text information of each independent component speech. The speech-to-text tool can use, for example, the existing technology Google Cloud Speech-to-Text tool. Then, the Chinese word segmentation tool is used to divide the text information of each independent component speech into several words, and its weight is determined according to the type of all words of all independent component speech, and the correlation between the word and the word in the predetermined matching word library is calculated, and the correlation of the word is multiplied by the corresponding weight to obtain the correlation. The greater the correlation, the greater the content correlation between the words in the text information of the independent component speech and the exhibition data. The independent component speech with the largest correlation is selected as the target independent component speech for interacting with the digital human, and the user corresponding to the target independent component speech is the interactive user interacting with the digital human. According to the above method, the target independent component speech corresponding to the sound collection device at all locations is obtained. Among them, the calculation method of the correlation can be the Pearson correlation algorithm, etc., and the specific method can be referred to the existing technology, and the embodiment of the present invention will not be repeated here.

[0046] S103, determining a similar speech group.

[0047] The similar speech group includes a first target independent component speech of a sound collecting device at a first position and a second target independent component speech of a sound collecting device at a second position.

[0048] Specifically, there are two cases for the target independent component voices collected by the sound collection devices at different positions: In the first case, the target independent component voices collected by the sound collection devices at different positions may come from the same sound source, in which case it is considered that there is only one target user participating in the interaction. In the second case, the target independent components collected by the sound collection devices at different positions correspond to different sound sources, in which case it is considered that there are multiple users participating in the interaction. The first target independent component voice of the sound collection device at the first position in the similar voice group and the second target independent component voice of the sound collection device at the second position have a high similarity. That is, the first target independent component voice of the sound collection device at the first position and the second target independent component voice of the sound collection device at the second position may come from the same sound source.

[0049] Further, when determining a similar speech group based on similarity, as an optional embodiment of the present invention, first determine a first average value of a first time interval between all adjacent maximum points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position, and a second average value of a second time interval between all adjacent minimum points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position; and determine a third average value of a third time interval between all adjacent maximum points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position, and a fourth average value of a fourth time interval between all adjacent minimum points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position; and determine the target independent component speech of the sound collection device at the first position at each moment. The method comprises the following steps: calculating a first difference between the sound frequency of the target independent component speech and the sound frequency of the target independent component speech of the sound collection device at the second position, and superimposing the first differences to obtain a superimposed value; then calculating a second sum of the first average value and the second average value, a third sum of the third average value and the fourth average value, and a second ratio of the second sum value to the third sum value; secondly calculating a second product of the absolute value of the second difference between the second ratio and the preset value and the superimposed value; then performing inverse proportional normalization on the second product to obtain the similarity between the target independent component speech of the sound collection device at the first position and the target independent component speech of the sound collection device at the second position; finally determining that the first target independent component speech of the sound collection device at the first position and the second target independent component speech of the sound collection device at the second position with the highest similarity are a similar speech group.

[0050] Specifically, in the embodiment of the present invention, the following formula can be used to calculate the similarity between the target independent component speech of the sound collection device at the first position and the target independent component speech of the sound collection device at the second position:

[0051]

[0052] In the above formula, represents the similarity between the t-th target independent component speech of the sound collection device at the first position and the k-th target independent component speech of the sound collection device at the second position, Represents a first average value of first time intervals between all adjacent maximum value points of the sound frequency sequence of the t-th target independent component speech of the sound collection device at the first position. Represents a second average value of second time intervals between all adjacent minimum value points of the sound frequency sequence of the t-th target independent component speech of the sound collecting device at the first position. Represents a third average value of the third time intervals between all adjacent maximum value points of the sound frequency sequence of the kth target independent component speech of the sound collection device at the second position. A fourth average value of a fourth time interval between all adjacent minimum value points of the sound frequency sequence of the kth target independent component speech of the sound collecting device at the second position. Indicates the maximum value of the collection time of the t-th target independent component speech of the sound collection device at the first position and the k-th target independent component speech of the sound collection device at the second position. Represents the sound frequency at the ith moment of the sound frequency sequence of the tth target independent component speech of the sound collection device at the first position. The sound frequency of the kth target independent component speech of the sound collection device at the second position at the ith moment. or If the value does not exist, it is 0. Denotes inverse normalization.

[0053] Furthermore, The closer the value of is to 1, the closer the extreme point interval between the t-th target independent component speech of the sound collection device at the first position and the k-th target independent component speech of the sound collection device at the second position is. The greater the similarity between the t-th target independent component speech of the sound collection device at the first position and the k-th target independent component speech of the sound collection device at the second position. The smaller the value, the greater the similarity. The greater the similarity, the more likely it is that the t-th target independent component speech of the sound collection device at the first position and the k-th target independent component speech of the sound collection device at the second position come from the same sound source.

[0054] The corresponding target independent component speech with the highest similarity between the t-th target independent component speech of the sound collection device at the first position and the k-th target independent component speech of the sound collection device at the second position is obtained and recorded as a similar speech group. A similar speech group includes target independent component speech representing a sound source at the first position and the second position.

[0055] S104, determining a distance difference between a first reference position of an interactive user to a first position and a second reference position of an interactive user to a second position according to a first acquisition time of a first target independent component speech, a second acquisition time of a second target independent component speech, and a sound propagation speed.

[0056] Specifically, the above-mentioned embodiment of the present invention performs independent component analysis on the voice collected by the sound collection devices at different locations, thereby identifying the voice content. The embodiment of the present invention also locates the position of the sound source in the target area to adjust the orientation of the digital human. In the embodiment of the present invention, the position where the digital human appears is preset to be the edge of the exhibition hall screen, so determining the orientation of the digital human only requires determining the direction of the interacting user. Taking the exhibition hall as an example, if the target area is the exhibition hall, Figure 3 As shown, Figure 3 A schematic top view of another exhibition hall provided by an embodiment of the present invention, Figure 3 In the embodiment of the present invention, since the position where the digital human appears is the screen at the edge of the exhibition hall, the polar coordinate system is established by taking the direction perpendicular to the screen toward the exhibition hall as the positive direction of the polar axis and determining the angle between the target position and the polar axis counterclockwise as the polar angle. Therefore, the positioning of the interactive user can be achieved by simply determining the value of the polar angle of the interactive user in the polar coordinate system. Further, Figure 3 As shown, it also includes a left reference point and a right reference point to facilitate more accurate positioning of the interactive user, wherein the left reference point is the first reference position in the embodiment of the present invention, the right reference point is the second reference position in the embodiment of the present invention, the left wall where the left reference point is located is the first position in the embodiment of the present invention, and the right wall where the right reference point is located is the second position in the embodiment of the present invention. Further, taking the second position as the right wall of the exhibition hall as an example, as Figure 4 As shown, Figure 4 A side view schematic diagram of a right wall provided in one embodiment of the present invention, the right reference point can be set at the center of the right wall. Similarly, the left reference point can also be set at the center of the left wall. It is worth noting that the first position, the second position, the first reference position and the second reference position can also be determined according to actual conditions, and the embodiment of the present invention is not limited here.

[0057] Further, when determining the distance difference, as an optional embodiment of the present invention, the first moment in the sound frequency sequence of the first target independent component speech when the sound frequency is greater than a predetermined value is first determined as the first collection moment; then the average value of the start moment of all the sound collection devices at the second position starting to collect the second target independent component speech is determined as the second collection moment; secondly, the difference between the first collection moment and the second collection moment is calculated to obtain the collection time; finally, the fourth product of the sound propagation speed and the collection time is determined as the distance difference.

[0058] Specifically, the predetermined value can be determined according to actual conditions, and in the embodiment of the present invention, the value is 100 Hz. In the embodiment of the present invention, the distance difference can be calculated using the following formula:

[0059]

[0060] In the above formula, Represents a distance difference between a first reference position from an interacting user to the first position and a second reference position from an interacting user to the second position. Represents the speed of sound propagation, usually taken as 340m / s in air at 15°C. The first moment when the sound frequency of the first target independent component speech of the similar speech group is greater than 100 Hz in the sound frequency sequence, that is, the first acquisition moment, It represents the average value of the start time when all the sound collection devices at the second position start to collect the second target independent component speech, that is, the second collection time.

[0061] Furthermore, When it is greater than 0, it means that the sound reaches the second position first and then the first position. The larger the value, the greater the difference, and the closer the sound source is to the second position.

[0062] S105: Determine a target position of the interactive user in the target area according to the distance between the first reference position and the second reference position and the distance difference.

[0063] Among them, when determining the target position of the interactive user, first determine the length of the line between the first reference position and the second reference position as the distance; then calculate the fourth sum of half the distance and the distance difference; secondly, take the first reference position as the starting point and take a target line segment with a length equal to the fourth sum on the line connecting the first reference position and the second reference position; finally determine the end point of the target line segment as the target position.

[0064] Specifically, the embodiment of the present invention combines the length of the line between the first reference position and the second reference position, takes the first reference position as the starting point, that is, the coordinate origin 0, and takes half of the length of the line on the line between the first reference position and the second reference position from the first reference position, and determines its length. That is, determine its The total length of the sum, starting from the first reference position, takes a target line segment of the total length on the line connecting the first reference position and the second reference position, the starting point of the target line segment is the first reference position, and the end point of the target line segment is the target position of the interactive user.

[0065] S106, using the target position of the interactive user in the target area, driving the digital human to face the interactive user.

[0066] Among them, when driving the digital human to face the interactive user, firstly, the vertex of the first position of the target area is taken as the pole, and the line between the vertex and the first reference position is taken as the polar axis; secondly, the angle between the line between the pole and the target position and the polar axis is determined as the polar angle of the interactive user; then, when there are multiple interactive users, the digital human is driven to face the interactive users at the corresponding target positions in order of polar angles from small to large; finally, when there is only one interactive user, the digital human is driven to face the interactive user at the target position. For example, when the first position is the left wall of the exhibition hall, the pole can be any vertex of the left wall.

[0067] The embodiment of the present invention can analyze the target independent component voice of the voice of the interactive user according to the audio data collected by the sound collection device, and then determine the similar voice group according to the target independent component voice, and use the target independent component voice of the sound collection device at different positions in the similar voice group and the reference position of each position to determine the target position of the interactive user interacting with the digital human. That is, the interactive user is positioned, and after the interactive user is positioned, the digital human is driven to face the target position where the interactive user is located. In this way, the orientation of the digital human can be flexibly adjusted according to the position of the interactive user participating in the interaction, which improves the flexibility of the interaction between the digital human and the audience, ensures the audience's sense of participation in the interactive process, and improves the audience's interactive experience.

[0068] Embodiment 2:

[0069] Corresponding to the digital human driving method based on voice recognition provided in the above embodiment, based on the same technical concept, the embodiment of the present invention further provides a digital human driving system based on voice recognition, and the digital human driving system based on voice recognition is used to execute the above digital human driving method based on voice recognition. Figure 5 A schematic diagram of a digital human driving system based on speech recognition is provided in one embodiment of the present invention. Figure 5As shown. The digital human driving system 500 based on speech recognition includes: an acquisition module 501, which is used to acquire audio data of each sound source in the target area through sound collection devices located at different positions in the target area; a determination module 502, which is used to determine the target independent component voice belonging to the voice of the interactive user from the audio data of each sound source; the determination module 502, which is also used to determine a similar voice group, the similar voice group includes a first target independent component voice of the sound collection device at the first position and a second target independent component voice of the sound collection device at the second position; the determination module 502, which is also used to determine the distance difference between the interactive user and the first reference position of the first position and the second reference position of the interactive user to the second position according to the first collection time of the first target independent component voice, the second collection time of the second target independent component voice and the sound propagation speed; the determination module 502 is also used to determine the target position of the interactive user in the target area according to the distance between the first reference position and the second reference position and the distance difference; the driving module 503, which is used to drive the digital human to face the interactive user using the target position of the interactive user in the target area.

[0070] The embodiment of the present invention can analyze the target independent component voice of the voice of the interactive user according to the audio data collected by the sound collection device, and then determine the similar voice group according to the target independent component voice, and use the target independent component voice of the sound collection device at different positions in the similar voice group and the reference position of each position to determine the target position of the interactive user interacting with the digital human. That is, the interactive user is positioned, and after the interactive user is positioned, the digital human is driven to face the target position where the interactive user is located. In this way, the orientation of the digital human can be flexibly adjusted according to the position of the interactive user participating in the interaction, which improves the flexibility of the interaction between the digital human and the audience, ensures the audience's sense of participation in the interactive process, and improves the audience's interactive experience.

[0071] Optionally, the determination module 502 is further used to perform independent component analysis on the audio data collected by the sound collection devices at different locations to obtain the sound frequency sequence of the independent component audio of each sound source at different times; determine the validity of the independent component audio according to the first number of extreme points in the sound frequency sequence, the first time interval between adjacent maximum points, the second time interval between adjacent minimum points, and the variance of the sound frequency in the sound frequency sequence; determine the independent component speech belonging to the speech based on the validity of the independent component audio; perform speech recognition on the independent component speech to obtain text information of the independent component speech; segment the text information to obtain multiple words; determine the interactive user who interacts with the digital human through the correlation between the words and the exhibition data corresponding to the digital human; determine the independent component speech of the interactive user as the target independent component speech.

[0072] Optionally, the determination module 502 is also used to determine a first average value of a first time interval between all adjacent maximum points of the sound frequency sequence, and a second average value of a second time interval between all adjacent minimum points of the sound frequency sequence; calculate a first sum of the first average value and the second average value, and calculate a first product between the first quantity and the variance; determine a first ratio between the first sum and the first product, and perform inverse proportional normalization processing on the first ratio to obtain the validity of the independent component audio.

[0073] Optionally, the determination module 502 is further configured to determine a similar speech group according to the similarity between the target independent component speech of the sound collection device at the first position of the target area and the target independent component speech of the sound collection device at the second position of the target area.

[0074] Optionally, the determination module 502 is further used to determine a first average value of a first time interval between all adjacent maximum points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position, a second average value of a second time interval between all adjacent minimum points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position; determine a third average value of a third time interval between all adjacent maximum points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position, a fourth average value of a fourth time interval between all adjacent minimum points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position; determine the sound frequency of the target independent component speech of the sound collection device at the first position at each moment The first difference between the sound frequency of the target independent component speech of the sound collection device at the first position and the sound collection device at the second position is calculated, and the first difference is superimposed to obtain a superimposed value; the second sum of the first average value and the second average value, the third sum of the third average value and the fourth average value, and the second ratio of the second sum value to the third sum value are calculated; the second product of the absolute value of the second difference between the second ratio and the preset value and the superimposed value is calculated; the second product is inversely normalized to obtain the similarity between the target independent component speech of the sound collection device at the first position and the target independent component speech of the sound collection device at the second position; the first target independent component speech of the sound collection device at the first position and the second target independent component speech of the sound collection device at the second position with the highest similarity are determined as a similar speech group.

[0075] Optionally, the determination module 502 is further used to determine the first moment in the sound frequency sequence of the first target independent component speech when the sound frequency is greater than a predetermined value as the first collection moment; determine the average value of the start moment of all the sound collection devices in the second position starting to collect the second target independent component speech as the second collection moment; calculate the difference between the first collection moment and the second collection moment to obtain the collection time; and determine the fourth product of the sound propagation speed and the collection time as the distance difference.

[0076] Optionally, the determination module 502 is also used to determine the length of the line between the first reference position and the second reference position as the distance; calculate the fourth sum of half the distance and the distance difference; take the first reference position as the starting point and take a target line segment on the line connecting the first reference position and the second reference position with a length equal to the fourth sum; and determine the end point of the target line segment as the target position.

[0077] Optionally, the driving module 503 is further used to take the vertex of the first position of the target area as the pole, and the line between the vertex and the first reference position as the polar axis; determine the angle between the line between the pole and the target position and the polar axis as the polar angle of the interactive user; when there are multiple interactive users, drive the digital human to face the interactive users at the corresponding target positions in order of polar angles from small to large; when there is only one interactive user, drive the digital human to face the interactive user at the target position.

[0078] Embodiment three:

[0079] Corresponding to the digital human driving method based on voice recognition provided in the above embodiment, based on the same technical concept, the embodiment of the present invention further provides a digital human driving device based on voice recognition, and the digital human driving device based on voice recognition is used to execute the above digital human driving method based on voice recognition. Figure 6 A schematic diagram of a digital human driving device based on speech recognition is provided in one embodiment of the present invention. Figure 6 The digital human driving system based on speech recognition may have relatively large differences due to different configurations or performances, and may include one or more processors 601 and memory 602, the memory 602 is used to store computer programs that can be run on the processor 601, and the processor 601 is used to execute the program stored in the memory 602 to achieve the above Figure 1 The memory 602 may be a temporary storage or a permanent storage. The application stored in the memory 602 may include one or more modules (not shown in the figure), each of which may include a series of computer executable instructions for the digital human driving system based on speech recognition.

[0080] Furthermore, the processor 601 can be configured to communicate with the memory 602 to execute a series of computer executable instructions in the memory 602 on the digital human driving system based on voice recognition. The digital human driving system based on voice recognition can also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input and output interfaces 605, and one or more keyboards 606.

[0081] Specifically in this embodiment, the digital human driving system based on voice recognition includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to achieve the above Figure 1 The various steps in the method embodiment are similar to those in the method embodiment, and have the beneficial effects of the above method embodiments. To avoid repetition, the embodiments of the present invention will not be described in detail here.

[0082] It should be noted that the digital human driving system based on voice recognition provided in the embodiment of the present invention and the digital human driving method based on voice recognition provided in the embodiment of the present invention are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned digital human driving method based on voice recognition, and has the same or similar beneficial effects, and the repeated parts will not be repeated.

[0083] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A digital human driving method based on speech recognition, characterized in that: The digital human driving method based on speech recognition comprises: Acquire audio data of each sound source in the target area through sound collection devices located at different positions in the target area; Determining a target independent component speech belonging to the speech of the interactive user from the audio data of each of the sound sources; Determine a similar speech group, wherein the similar speech group includes a first target independent component speech of a sound collection device at a first position and a second target independent component speech of a sound collection device at a second position; Determine, according to a first acquisition time of the first target independent component speech, a second acquisition time of the second target independent component speech, and a sound propagation speed, a distance difference between a first reference position of the interactive user to the first position and a second reference position of the interactive user to the second position; determining a target position of the interactive user within the target area according to the distance between the first reference position and the second reference position and the distance difference; Using the target position of the interactive user in the target area, the digital human is driven to face the interactive user; The step of determining the target independent component voice belonging to the voice of the interactive user from the audio data of each of the sound sources comprises: Performing independent component analysis on the audio data collected by the sound collection devices at different locations to obtain sound frequency sequences of the independent component audio of each sound source at different times; Determining the validity of the independent component audio according to a first number of extreme value points in the sound frequency sequence, a first time interval between adjacent maximum value points, a second time interval between adjacent minimum value points, and a variance of the sound frequency in the sound frequency sequence; Determining the independent component audio that belongs to the speech based on the validity of the independent component audio; Performing speech recognition on the independent component speech to obtain text information of the independent component speech; Segmenting the text information to obtain multiple words; Determine an interactive user interacting with the digital human through the correlation between the vocabulary and the exhibition data corresponding to the digital human; The independent component speech of the interactive user is determined as the target independent component speech.

2. The digital human driving method based on speech recognition according to claim 1 is characterized in that: Determining the validity of the independent component audio according to the first number of extreme value points in the sound frequency sequence, the first time interval between adjacent maximum value points, the second time interval between adjacent minimum value points, and the variance of the sound frequency in the sound frequency sequence includes: Determine a first average value of a first time interval between all adjacent maximum value points of the sound frequency sequence, and a second average value of a second time interval between all adjacent minimum value points of the sound frequency sequence; calculating a first sum of the first average value and the second average value, and calculating a first product between the first quantity and the variance; A first ratio between the first sum and the first product is determined, and an inverse proportional normalization process is performed on the first ratio to obtain the validity of the independent component audio.

3. The digital human driving method based on speech recognition according to claim 1, characterized in that: The sound collection device includes a sound collection device located at a first position of the target area and a sound collection device located at a second position of the target area, and determining the similar voice group includes: The similar speech group is determined according to the similarity between the target independent component speech of the sound collecting device at the first position of the target area and the target independent component speech of the sound collecting device at the second position of the target area.

4. The digital human driving method based on speech recognition according to claim 3 is characterized in that: The determining the similar speech group according to the similarity between the target independent component speech of the sound collection device at the first position of the target area and the target independent component speech of the sound collection device at the second position of the target area comprises: Determine a first average value of a first time interval between all adjacent maximum value points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position, and a second average value of a second time interval between all adjacent minimum value points of the sound frequency sequence of the target independent component speech of the sound collection device at the first position; Determine a third average value of a third time interval between all adjacent maximum value points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position, and a fourth average value of a fourth time interval between all adjacent minimum value points of the sound frequency sequence of the target independent component speech of the sound collection device at the second position; Determine a first difference between the sound frequency of the target independent component speech of the sound collection device at the first position and the sound frequency of the target independent component speech of the sound collection device at the second position at each moment, and superimpose the first differences to obtain a superimposed value; calculating a second sum of the first average value and the second average value, a third sum of the third average value and the fourth average value, and a second ratio of the second sum value to the third sum value; Calculating a second product of an absolute value of a second difference between the second ratio and a preset value and the superposition value; Performing inverse proportional normalization on the second product to obtain the similarity between the target independent component speech of the sound collection device at the first position and the target independent component speech of the sound collection device at the second position; The first target independent component speech of the sound collecting device at the first position and the second target independent component speech of the sound collecting device at the second position with the highest similarity are determined as the similar speech group.

5. The digital human driving method based on voice recognition according to any one of claims 1 to 4, characterized in that: The determining, according to the first acquisition time of the first target independent component speech, the second acquisition time of the second target independent component speech, and the sound propagation speed, a distance difference between the first reference position of the interactive user to the first position and the second reference position of the interactive user to the second position comprises: Determine the first moment in the sound frequency sequence of the first target independent component speech when the sound frequency is greater than a predetermined value as the first acquisition moment; Determine the average value of the start time of all the sound collection devices at the second position starting to collect the second target independent component speech as the second collection time; Calculate the difference between the first collection time and the second collection time to obtain a collection duration; A fourth product of the sound propagation speed and the collection time is determined as the distance difference.

6. The digital human driving method based on voice recognition according to any one of claims 1 to 4, characterized in that: Determining the target position of the interactive user in the target area according to the distance between the first reference position and the second reference position and the distance difference comprises: Determine the length of a line between the first reference position and the second reference position as the distance; calculating a fourth sum of half the distance and the distance difference; Taking the first reference position as a starting point, taking a target line segment having a length equal to the fourth sum value on a line connecting the first reference position and the second reference position; The end point of the target line segment is determined as the target position.

7. The digital human driving method based on voice recognition according to any one of claims 1 to 4, characterized in that: The step of using the interactive user at the target position within the target area to drive the digital human to face the interactive user comprises: Taking the vertex of the first position of the target area as the pole, and the line between the vertex and the first reference position as the polar axis; Determine the angle between the line between the pole and the target position and the polar axis as the polar angle of the interactive user; In the case that there are multiple interactive users, the digital human is driven to face the interactive users at corresponding target positions in order of the polar angles from small to large; When there is only one interactive user, the digital human is driven to face the interactive user at the target location.

8. A digital human driving system based on speech recognition, characterized in that: include: An acquisition module, used to acquire audio data of each sound source in the target area through sound collection devices located at different positions in the target area; A determination module, used to determine the target independent component speech belonging to the speech of the interactive user from the audio data of each of the sound sources; The determination module is further used to determine a similar speech group, wherein the similar speech group includes a first target independent component speech of a sound collection device at a first position and a second target independent component speech of a sound collection device at a second position; The determination module is further used to determine a distance difference between a first reference position of the interactive user to the first position and a second reference position of the interactive user to the second position according to a first acquisition time of the first target independent component speech, a second acquisition time of the second target independent component speech, and a sound propagation speed; The determination module is further configured to determine a target position of the interactive user within the target area according to a distance between the first reference position and the second reference position and the distance difference; A driving module, used for driving the digital human to face the interactive user using the target position of the interactive user in the target area; The step of determining the target independent component voice belonging to the voice of the interactive user from the audio data of each of the sound sources comprises: Performing independent component analysis on the audio data collected by the sound collection devices at different locations to obtain sound frequency sequences of the independent component audio of each sound source at different times; Determining the validity of the independent component audio according to a first number of extreme value points in the sound frequency sequence, a first time interval between adjacent maximum value points, a second time interval between adjacent minimum value points, and a variance of the sound frequency in the sound frequency sequence; Determining the independent component audio that belongs to the speech based on the validity of the independent component audio; Performing speech recognition on the independent component speech to obtain text information of the independent component speech; Segmenting the text information to obtain multiple words; Determine an interactive user interacting with the digital human through the correlation between the vocabulary and the exhibition data corresponding to the digital human; The independent component speech of the interactive user is determined as the target independent component speech.

9. A digital human driving device based on voice recognition, characterized in that: The digital human driving device based on voice recognition includes: a processor and a memory; wherein the memory is used to store computer programs that can be run on the processor; the processor is used to execute the program stored in the memory to implement the steps of the digital human driving method based on voice recognition as described in any one of claims 1-7.

Images