Digital human interaction method and system based on voice recognition

Abstract

The application discloses a digital human interaction method and system based on voice recognition, and the method comprises the following steps: obtaining a plurality of historical input voices input by an interaction object to a digital human object; determining a multidimensional vector corresponding to all the historical input voices based on a multidimensional vector extraction algorithm; predicting a context parameter corresponding to the interaction object and the digital human object according to the multidimensional vector; determining an interaction rule of the digital human object relative to the interaction object according to a prediction model corresponding to the context parameter and the historical input voices; and the interaction rule is used for defining a data rule followed by a response voice and / or a response action of the digital human object. It can be seen that the application can improve the semantic understanding depth of the digital human to the interaction environment and the automation degree of the response logic, and optimize the precision of the data feedback and the reliability of the system decision in the human-computer interaction process.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a digital human interaction method and system based on speech recognition. Background Technology

[0002] As digital human technology evolves from simple visual representation to deep semantic interaction, enabling digital humans to "understand" implied meanings and provide feedback appropriate to the current context has become crucial for enhancing the realism of interactions. However, existing digital human interaction technologies typically rely on pre-defined keyword matching or generic dialogue scripts. The lack of technical support for acquiring historical input speech from the interacting object and using multi-dimensional vector prediction of contextual parameters makes it impossible to dynamically determine the interaction rules of the digital human object for specific interaction scenarios. This hinders the achievement of deep semantic understanding of the interaction environment and automated processing of response logic, resulting in insufficient accuracy of data feedback and reliability of system decisions during human-computer interaction. This limits the intelligent performance and user experience of digital humans in complex and ever-changing instant messaging or service scenarios. Clearly, existing technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a digital human interaction method and system based on speech recognition, which can improve the depth of digital human's semantic understanding of the interactive environment and the degree of automation of response logic, and optimize the accuracy of data feedback and the reliability of system decision-making during human-computer interaction.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a digital human interaction method based on speech recognition, the method comprising: Acquire multiple historical voice inputs from the interactive object to the digital human object; Based on a multi-dimensional vector extraction algorithm, determine the multi-dimensional vectors corresponding to all the historical input speech; Based on the multi-dimensional vector, predict the contextual parameters corresponding to the interactive object and the digital human object; Based on the prediction model corresponding to the context parameters and the historical input speech, the interaction rules of the digital human object relative to the interactive object are determined; the interaction rules are used to limit the data rules followed by the response speech and / or response actions of the digital human object.

[0005] As an optional implementation, in the first aspect of the invention, determining the multi-dimensional vectors corresponding to all the historical input speech based on the multi-dimensional vector extraction algorithm includes: For each historical input speech, the historical input speech is input into a multi-dimensional parameter prediction model to obtain the parameters of multiple dimensions corresponding to the historical input speech; Based on the vectorization model corresponding to each parameter, the historical input speech is vectorized to obtain the speech parameter vector; The speech parameter vectors of all the historical input speech are assembled to obtain the corresponding multi-dimensional vector.

[0006] As an optional implementation, in the first aspect of the present invention, the dimension is a spatial dimension or a temporal dimension; the spatial dimension includes at least one of an echo dimension, a distance dimension, and a location dimension; the temporal dimension includes at least one of a speech rate dimension, a volume dimension, and a duration dimension.

[0007] As an optional implementation, in the first aspect of the invention, the step of vectorizing the historical input speech according to the vectorization model corresponding to each parameter to obtain a speech parameter vector includes: For each parameter, the corresponding vectorized model is matched among multiple candidate vectorized models; The historical input speech is input into the vectorized model to obtain the parameter-related vector corresponding to the parameter; Based on the preset parameter structure corresponding to all the parameters, the speech parameter vector corresponding to the historical input speech is obtained by assembling all the parameter-related vectors.

[0008] As an optional implementation, in the first aspect of the invention, assembling the speech parameter vectors of all the historical input speech to obtain a corresponding multi-dimensional vector includes: Calculate the vector similarity between the speech parameter vectors of any two historical input speech samples; Each historical input speech is defined as a graph node, and the corresponding speech parameter vector is defined as the node feature value of the graph node. The similarity of the vectors corresponding to any two graph nodes is determined as the node association feature between the two graph nodes; All the graph nodes and their corresponding node feature values ​​and node association features are determined as corresponding multi-dimensional vectors.

[0009] As an optional implementation, in the first aspect of the present invention, predicting the contextual parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector includes: The multi-dimensional vectors are input into the trained context prediction model to obtain the context parameters corresponding to the interactive object and the digital human object; the context prediction model is trained using a training dataset that includes multiple training dialogue records and corresponding multi-dimensional vector annotations and context annotations.

[0010] As an optional implementation, in the first aspect of the present invention, the context parameters include at least one of dialogue space type, dialogue space area, interaction purpose, interaction friendliness, and understanding level.

[0011] As an optional implementation, in the first aspect of the present invention, determining the interaction rules of the digital human object relative to the interactive object based on the prediction model corresponding to the context parameters and the historical input speech includes: Based on the correspondence between the scenario parameters and the preset first parameter, the motion restriction parameters of the digital human object are determined; the motion restriction parameters include at least one of amplitude restriction parameters, movement distance restriction parameters, swing angle restriction parameters, and movement part restriction parameters; Based on the correspondence between the context parameters and the preset second parameters, the voice restriction parameters of the digital human object are determined; the voice restriction parameters include at least one of the following: length restriction parameters, tone restriction parameters, voice-text restriction parameters, and voice volume restriction parameters. Generate a pre-cue word rule that includes the action restriction parameters and the speech restriction parameters; The pre-prompt word rule is connected to the data input interface of the multimodal model of the digital human object, so that each prediction data input is accompanied by a prompt word processed by the pre-prompt word rule to correct the output speech and / or output motion image of the multimodal model.

[0012] A second aspect of this invention discloses a digital human interaction system based on speech recognition, the system comprising: The acquisition module is used to acquire multiple historical voice inputs from the interactive object to the digital human object; The extraction module is used to determine the multi-dimensional vectors corresponding to all the historical input speech based on a multi-dimensional vector extraction algorithm; The prediction module is used to predict the context parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector. The determination module is used to determine the interaction rules of the digital human object relative to the interactive object based on the prediction model corresponding to the context parameters and the historical input voice; the interaction rules are used to limit the data rules followed by the response voice and / or response actions of the digital human object.

[0013] As an optional implementation, in a second aspect of the invention, the extraction module determines the specific method for determining the multi-dimensional vectors corresponding to all the historical input speech based on a multi-dimensional vector extraction algorithm, including: For each historical input speech, the historical input speech is input into a multi-dimensional parameter prediction model to obtain the parameters of multiple dimensions corresponding to the historical input speech; Based on the vectorization model corresponding to each parameter, the historical input speech is vectorized to obtain the speech parameter vector; The speech parameter vectors of all the historical input speech are assembled to obtain the corresponding multi-dimensional vector.

[0014] As an optional implementation, in the second aspect of the present invention, the dimension is a spatial dimension or a temporal dimension; the spatial dimension includes at least one of an echo dimension, a distance dimension, and a location dimension; the temporal dimension includes at least one of a speech rate dimension, a volume dimension, and a duration dimension.

[0015] As an optional implementation, in a second aspect of the invention, the extraction module vectorizes the historical input speech according to the vectorization model corresponding to each parameter to obtain a speech parameter vector, including: For each parameter, the corresponding vectorized model is matched among multiple candidate vectorized models; The historical input speech is input into the vectorized model to obtain the parameter-related vector corresponding to the parameter; Based on the preset parameter structure corresponding to all the parameters, the speech parameter vector corresponding to the historical input speech is obtained by assembling all the parameter-related vectors.

[0016] As an optional implementation, in a second aspect of the invention, the extraction module assembles the speech parameter vectors of all the historical input speech to obtain the corresponding multi-dimensional vector in the following specific manner: Calculate the vector similarity between the speech parameter vectors of any two historical input speech samples; Each historical input speech is defined as a graph node, and the corresponding speech parameter vector is defined as the node feature value of the graph node. The similarity of the vectors corresponding to any two graph nodes is determined as the node association feature between the two graph nodes; All the graph nodes and their corresponding node feature values ​​and node association features are determined as corresponding multi-dimensional vectors.

[0017] As an optional implementation, in a second aspect of the invention, the specific method by which the prediction module predicts the contextual parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector includes: The multi-dimensional vectors are input into the trained context prediction model to obtain the context parameters corresponding to the interactive object and the digital human object; the context prediction model is trained using a training dataset that includes multiple training dialogue records and corresponding multi-dimensional vector annotations and context annotations.

[0018] As an optional implementation, in a second aspect of the invention, the contextual parameters include at least one of dialogue space type, dialogue space area, interaction purpose, interaction friendliness, and understanding level.

[0019] As an optional implementation, in a second aspect of the invention, the determining module determines the specific manner of the interaction rules between the digital human object and the interactive object based on the prediction model corresponding to the context parameters and the historical input speech, including: Based on the correspondence between the scenario parameters and the preset first parameter, the motion restriction parameters of the digital human object are determined; the motion restriction parameters include at least one of amplitude restriction parameters, movement distance restriction parameters, swing angle restriction parameters, and movement part restriction parameters; Based on the correspondence between the context parameters and the preset second parameters, the voice restriction parameters of the digital human object are determined; the voice restriction parameters include at least one of the following: length restriction parameters, tone restriction parameters, voice-text restriction parameters, and voice volume restriction parameters. Generate a pre-cue word rule that includes the action restriction parameters and the speech restriction parameters; The pre-prompt word rule is connected to the data input interface of the multimodal model of the digital human object, so that each prediction data input is accompanied by a prompt word processed by the pre-prompt word rule to correct the output speech and / or output motion image of the multimodal model.

[0020] A third aspect of the present invention discloses another digital human interaction system based on speech recognition, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the speech recognition-based digital human interaction method disclosed in the first aspect of the present invention.

[0021] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the speech recognition-based digital human interaction method disclosed in the first aspect of the present invention.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention acquires the historical input speech of the interactive object and extracts multi-dimensional vectors based on a multi-dimensional vector extraction algorithm. Then, it uses the multi-dimensional vectors to predict contextual parameters and determine the interaction rules of the digital human object. This enables the digital human to improve the semantic understanding depth of the interactive environment and the automation level of the response logic, and optimizes the accuracy of data feedback and the reliability of system decision-making during human-computer interaction. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a digital human interaction method based on speech recognition disclosed in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of a digital human interaction system based on speech recognition disclosed in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of another digital human interaction system based on speech recognition disclosed in an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] This invention discloses a digital human interaction method and system based on speech recognition. By acquiring the historical input speech of the interaction object and extracting multi-dimensional vectors using a multi-dimensional vector extraction algorithm, the method then uses these multi-dimensional vectors to predict contextual parameters and determine the interaction rules for the digital human object. This improves the depth of the digital human's semantic understanding of the interaction environment and the automation level of its response logic, optimizing the accuracy of data feedback and the reliability of system decisions during human-computer interaction. Detailed explanations follow.

[0031] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a digital human interaction method based on speech recognition disclosed in an embodiment of the present invention. Figure 1 The described speech recognition-based digital human interaction method can be applied to data processing systems / data processing devices / data processing servers (wherein, the server includes a local processing server or a cloud processing server). For example... Figure 1 As shown, this speech recognition-based digital human interaction method may include the following operations: 101. Obtain multiple historical voice inputs from the interactive object to the digital human object.

[0032] Optionally, the historical input voice can be PCM format audio, WAV format audio, or MP3 format audio recorded through an omnidirectional microphone array, MEMS microphone, bone conduction headphones, etc., and the present invention does not limit it.

[0033] 102. Based on the multi-dimensional vector extraction algorithm, determine the multi-dimensional vectors corresponding to all historical input speech.

[0034] Optionally, the multi-dimensional vector extraction algorithm can be a Mel frequency cepstral coefficient (MFCC) extraction algorithm, a perceptual linear prediction (PLP) algorithm, a deep convolutional neural network (CNN) feature extraction algorithm, or a Transformer-based self-supervised speech representation learning algorithm; this invention does not limit the algorithm.

[0035] 103. Based on multi-dimensional vectors, predict the contextual parameters corresponding to interactive objects and digital human objects.

[0036] 104. Based on the prediction model corresponding to the context parameters and the historical input speech, determine the interaction rules of the digital human object relative to the interactive object.

[0037] Optionally, interaction rules are used to define the data rules followed by the digital human object's response speech and / or response actions.

[0038] As can be seen, the above-described embodiments of the invention obtain the historical input speech of the interactive object and extract multi-dimensional vectors based on a multi-dimensional vector extraction algorithm, and then use the multi-dimensional vectors to predict contextual parameters and determine the interaction rules of the digital human object. This enables the digital human to improve the semantic understanding depth of the interactive environment and the automation level of the response logic, and optimize the accuracy of data feedback and the reliability of system decision-making during human-computer interaction.

[0039] As an optional embodiment, the step above, determining the multi-dimensional vectors corresponding to all historical input speech based on the multi-dimensional vector extraction algorithm, includes: For each historical input speech, the historical input speech is input into a multi-dimensional parameter prediction model to obtain the parameters of multiple dimensions corresponding to the historical input speech; Based on the vectorization model corresponding to each parameter, the historical input speech is vectorized to obtain the speech parameter vector; The speech parameter vectors of all historical input speech are assembled to obtain the corresponding multi-dimensional vector.

[0040] Optional, the dimension can be either spatial or temporal.

[0041] Optionally, the spatial dimension includes at least one of the echo dimension, distance dimension, and orientation dimension.

[0042] Optionally, the time dimension includes at least one of the speech rate dimension, volume dimension, and duration dimension.

[0043] Optionally, the orientation dimension can be the horizontal azimuth angle, vertical pitch angle, or three-dimensional coordinates of the sound source in a spherical coordinate system determined based on the time difference of arrival (TDOA). This invention does not impose any limitations on this.

[0044] As can be seen, through the above optional embodiments, multiple feature parameters of speech in the spatial and temporal dimensions are extracted by the multi-dimensional parameter prediction model and assembled into a multi-dimensional vector, realizing a high-fine-grained digital deconstruction of the original audio signal, greatly reducing the information entropy loss in the feature extraction process, thereby providing more discriminative input features for subsequent context recognition algorithms, and effectively improving the ability of the computing model to analyze dynamic interactive scenarios.

[0045] As an optional embodiment, the step described above, vectorizing the historical input speech according to the vectorization model corresponding to each parameter to obtain a speech parameter vector, includes: For each parameter, the corresponding vectorized model is matched among multiple candidate vectorized models; The historical input speech is fed into the vectorized model to obtain the parameter-related vector corresponding to that parameter; The speech parameter vector corresponding to the historical input speech is obtained by assembling all parameter-related vectors based on the preset parameter structure corresponding to all parameters.

[0046] Optionally, the parameter-related vector can be a one-hot encoded vector, a dense embedding vector with fixed dimensions, a normalized floating-point tensor, or a principal component feature vector after dimensionality reduction. This invention does not impose any limitations.

[0047] As can be seen, through the above optional embodiments, by matching corresponding candidate vectorization models for parameters of different dimensions and assembling vectors based on preset parameter structures, the optimal mapping processing can be achieved for feature data with different physical meanings, ensuring the consistency of heterogeneous parameters in numerical expression and the rigor of logical structure, thereby improving the system's processing efficiency and the stability of multimodal data fusion in a large-scale data throughput environment.

[0048] As an optional embodiment, the step above, assembling the speech parameter vectors of all historical input speech to obtain the corresponding multi-dimensional vector, includes: Calculate the vector similarity between the speech parameter vectors of any two historical input speech samples; Each historical input speech is defined as a graph node, and the corresponding speech parameter vector is defined as the node feature value of the graph node. The similarity of vectors corresponding to any two graph nodes is determined as the node association feature between the two graph nodes. All graph nodes, their corresponding node feature values, and node association features are determined as corresponding multi-dimensional vectors.

[0049] Optionally, the vector similarity can be cosine similarity, reciprocal of Euclidean distance, Mahalanobis distance, Manhattan distance, or Chebyshev distance; this invention does not limit the type of similarity.

[0050] Optionally, the node association feature can be represented as an adjacency matrix, a Laplacian matrix, a weighted graph edge set, or an attention weight distribution matrix; this invention does not impose any limitations on this.

[0051] As can be seen, through the above optional embodiments, by calculating the similarity between historical speech vectors and constructing a graph structure with vectors as node features and similarity as association features, discrete speech segments are successfully transformed into a data network with topological associations. The ability of the graph model to capture deep implicit relationships greatly enriches the information density of the feature vectors, thereby enabling the system to have higher computational accuracy and generalization performance when recognizing the behavior patterns of interactive objects.

[0052] As an optional embodiment, the step above, predicting the contextual parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector, includes: Multi-dimensional vectors are input into the trained context prediction model to obtain context parameters corresponding to the interactive objects and digital human objects.

[0053] Optionally, the context prediction model is trained using a training dataset that includes multiple training dialogue records and corresponding multi-dimensional vector annotations and context annotations.

[0054] Optionally, the scenario prediction model can be a gradient boosting decision tree (GBDT), a random forest model, a deep neural network (DNN), or a graph convolutional network (GCN), and this invention does not limit it.

[0055] Optionally, the contextual parameters include at least one of the following: dialogue space type, dialogue space area, interaction purpose, interaction friendliness, and level of understanding.

[0056] Optionally, the dialogue space type can be a private bedroom, a public square, an office meeting room, a noisy restaurant, an open area of ​​a shopping mall, or a quiet library; this invention does not limit the type of space.

[0057] As can be seen, through the above optional embodiments, by inputting multi-dimensional vectors into the trained context prediction model to identify context parameters such as dialogue space and interaction intent, automatic mapping from high-dimensional abstract features to specific business logic parameters is achieved. The non-linear fitting capability of deep learning models is used to significantly improve the accuracy of perception of the interactive environment, providing real-time and standardized data support for the adaptive adjustment of digital human systems.

[0058] As an optional embodiment, the step above, determining the interaction rules between the digital human object and the interactive object based on the prediction model corresponding to the context parameters and the historical input speech, includes: Based on the correspondence between the scenario parameters and the preset first parameter, determine the action restriction parameters of the digital human object; Based on the correspondence between the contextual parameters and the preset second parameter, determine the voice restriction parameters of the digital human object; Generate pre-cue word rules that include action restriction parameters and speech restriction parameters; The pre-cue word rules are connected to the data input interface of the multimodal model of the digital human object so that each prediction data input is accompanied by a cue word processed by the pre-cue word rules to correct the output speech and / or output motion image of the multimodal model.

[0059] Optionally, the motion restriction parameters include at least one of the following: amplitude restriction parameters, movement distance restriction parameters, swing angle restriction parameters, and motion part restriction parameters.

[0060] Optionally, the voice restriction parameters include at least one of the following: length restriction parameters, tone restriction parameters, voice-text restriction parameters, and voice volume restriction parameters.

[0061] Optionally, the action part restriction parameter can be a facial expression weight threshold, a range of limb joint degrees of freedom constraint, a gesture interaction frequency or a stride width restriction value; this invention does not impose any limitations on this parameter.

[0062] Optionally, the tone restriction parameter can be an emotion tendency coefficient, a range of intonation fluctuation frequency, an accent position marker, or an emotion classification label (such as friendly, solemn, or humorous), and this invention does not impose any limitations on it.

[0063] Optionally, the multimodal model can be a video generation model based on a diffusion model, an action generation network based on a generative adversarial network, a CLIP cross-modal contrastive learning model, or a language model with visual / speech understanding capabilities (such as GPT-4o, Gemini). This invention does not limit the scope of the model.

[0064] As can be seen, through the above optional embodiments, by mapping contextual parameters to action and speech constraint parameters and generating pre-prompt word rules to correct the multimodal model output, a real-time closed-loop control mechanism based on contextual constraints is constructed at the algorithm level. This effectively solves the problem of uncontrollability of the output content of generative models, thereby significantly enhancing the computational synergy efficiency between digital human expressiveness and environmental fit while ensuring output quality.

[0065] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a digital human interaction system based on speech recognition disclosed in an embodiment of the present invention. Figure 2 The described speech recognition-based digital human interaction system can be applied to data processing systems / data processing equipment / data processing servers (wherein, the server includes local processing servers or cloud processing servers). For example... Figure 2 As shown, the speech recognition-based digital human interaction system may include: The acquisition module 201 is used to acquire multiple historical input voices from the interactive object to the digital human object.

[0066] Extraction module 202 is used to determine the multi-dimensional vectors corresponding to all historical input speech based on a multi-dimensional vector extraction algorithm.

[0067] The prediction module 203 is used to predict the context parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector.

[0068] The determination module 204 is used to determine the interaction rules of the digital human object relative to the interactive object based on the prediction model corresponding to the context parameters and the historical input speech.

[0069] Optionally, interaction rules are used to define the data rules followed by the digital human object's response speech and / or response actions.

[0070] As can be seen, the above-described embodiments of the invention obtain the historical input speech of the interactive object and extract multi-dimensional vectors based on a multi-dimensional vector extraction algorithm, and then use the multi-dimensional vectors to predict contextual parameters and determine the interaction rules of the digital human object. This enables the digital human to improve the semantic understanding depth of the interactive environment and the automation level of the response logic, and optimize the accuracy of data feedback and the reliability of system decision-making during human-computer interaction.

[0071] As an optional embodiment, the extraction module determines the specific method of the multi-dimensional vector corresponding to all historical input speech based on a multi-dimensional vector extraction algorithm, including: For each historical input speech, the historical input speech is input into a multi-dimensional parameter prediction model to obtain the parameters of multiple dimensions corresponding to the historical input speech; Based on the vectorization model corresponding to each parameter, the historical input speech is vectorized to obtain the speech parameter vector; The speech parameter vectors of all historical input speech are assembled to obtain the corresponding multi-dimensional vector.

[0072] As can be seen, through the above optional embodiments, multiple feature parameters of speech in the spatial and temporal dimensions are extracted by the multi-dimensional parameter prediction model and assembled into a multi-dimensional vector, realizing a high-fine-grained digital deconstruction of the original audio signal, greatly reducing the information entropy loss in the feature extraction process, thereby providing more discriminative input features for subsequent context recognition algorithms, and effectively improving the ability of the computing model to analyze dynamic interactive scenarios.

[0073] As an optional embodiment, the dimension is a spatial dimension or a temporal dimension; the spatial dimension includes at least one of the reverberation dimension, distance dimension, and orientation dimension; the temporal dimension includes at least one of the speech rate dimension, volume dimension, and duration dimension.

[0074] As can be seen, the above optional embodiments limit the types of voice dimensions to comprehensively represent the multi-dimensional characteristics of user voice, and help to improve the depth of digital human's semantic understanding of the interactive environment and the degree of automation of response logic.

[0075] As an optional embodiment, the extraction module vectorizes the historical input speech according to the vectorization model corresponding to each parameter to obtain the speech parameter vector in the following specific ways: For each parameter, the corresponding vectorized model is matched among multiple candidate vectorized models; The historical input speech is fed into the vectorized model to obtain the parameter-related vector corresponding to that parameter; The speech parameter vector corresponding to the historical input speech is obtained by assembling all parameter-related vectors based on the preset parameter structure corresponding to all parameters.

[0076] As can be seen, through the above optional embodiments, by matching corresponding candidate vectorization models for parameters of different dimensions and assembling vectors based on preset parameter structures, the optimal mapping processing can be achieved for feature data with different physical meanings, ensuring the consistency of heterogeneous parameters in numerical expression and the rigor of logical structure, thereby improving the system's processing efficiency and the stability of multimodal data fusion in a large-scale data throughput environment.

[0077] As an optional embodiment, the extraction module assembles the speech parameter vectors of all historical input speech to obtain the corresponding multi-dimensional vector in the following specific ways: Calculate the vector similarity between the speech parameter vectors of any two historical input speech samples; Each historical input speech is defined as a graph node, and the corresponding speech parameter vector is defined as the node feature value of the graph node. The similarity of vectors corresponding to any two graph nodes is determined as the node association feature between the two graph nodes. All graph nodes, their corresponding node feature values, and node association features are determined as corresponding multi-dimensional vectors.

[0078] As can be seen, through the above optional embodiments, by calculating the similarity between historical speech vectors and constructing a graph structure with vectors as node features and similarity as association features, discrete speech segments are successfully transformed into a data network with topological associations. The ability of the graph model to capture deep implicit relationships greatly enriches the information density of the feature vectors, thereby enabling the system to have higher computational accuracy and generalization performance when recognizing the behavior patterns of interactive objects.

[0079] As an optional embodiment, the prediction module predicts the specific methods by which it predicts the contextual parameters corresponding to the interactive object and the digital human object based on multi-dimensional vectors, including: Multi-dimensional vectors are input into the trained context prediction model to obtain context parameters corresponding to the interactive object and the digital human object; the context prediction model is trained using a training dataset that includes multiple training dialogue records and corresponding multi-dimensional vector annotations and context annotations.

[0080] As can be seen, through the above optional embodiments, by inputting multi-dimensional vectors into the trained context prediction model to identify context parameters such as dialogue space and interaction intent, automatic mapping from high-dimensional abstract features to specific business logic parameters is achieved. The non-linear fitting capability of deep learning models is used to significantly improve the accuracy of perception of the interactive environment, providing real-time and standardized data support for the adaptive adjustment of digital human systems.

[0081] As an optional embodiment, the context parameters include at least one of the following: dialogue space type, dialogue space area, interaction purpose, interaction friendliness, and understanding level.

[0082] As can be seen, the content of the context parameters is limited through the above optional embodiments to comprehensively characterize the multi-dimensional characteristics of the interaction context between the user and the digital human object, and to help improve the depth of the digital human's semantic understanding of the interaction environment and the degree of automation of the response logic.

[0083] As an optional embodiment, the determining module determines the specific manner of the interaction rules between the digital human object and the interactive object based on the prediction model corresponding to the context parameters and the historical input speech, including: Based on the correspondence between the scenario parameters and the preset first parameter, the motion restriction parameters of the digital human object are determined; optionally, the motion restriction parameters include at least one of the following: amplitude restriction parameters, movement distance restriction parameters, swing angle restriction parameters, and movement part restriction parameters. Based on the correspondence between the context parameters and the preset second parameter, the voice restriction parameters of the digital human object are determined; optionally, the voice restriction parameters include at least one of the following: length restriction parameters, tone restriction parameters, voice text restriction parameters, and voice volume restriction parameters. Generate pre-cue word rules that include action restriction parameters and speech restriction parameters; The pre-cue word rules are connected to the data input interface of the multimodal model of the digital human object so that each prediction data input is accompanied by a cue word processed by the pre-cue word rules to correct the output speech and / or output motion image of the multimodal model.

[0084] As can be seen, through the above optional embodiments, by mapping contextual parameters to action and speech constraint parameters and generating pre-prompt word rules to correct the multimodal model output, a real-time closed-loop control mechanism based on contextual constraints is constructed at the algorithm level. This effectively solves the problem of uncontrollability of the output content of generative models, thereby significantly enhancing the computational synergy efficiency between digital human expressiveness and environmental fit while ensuring output quality.

[0085] Example 3 Please see Figure 3 , Figure 3 This is another digital human interaction system based on speech recognition disclosed in the embodiments of the present invention. Figure 3 The described speech recognition-based digital human interaction system is applied in data processing systems / data processing equipment / data processing servers (wherein, the server includes a local processing server or a cloud processing server). For example... Figure 3 As shown, the speech recognition-based digital human interaction system may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the speech recognition-based digital human interaction method described in Embodiment 1.

[0086] Example 4 This invention discloses a computer read storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the steps of the speech recognition-based digital human interaction method described in Embodiment 1.

[0087] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the speech recognition-based digital human interaction method described in Embodiment 1.

[0088] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0090] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0091] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0096] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0097] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0098] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0099] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0100] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0101] Finally, it should be noted that the digital human interaction method and system based on speech recognition disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital human interaction method based on speech recognition, characterized in that, The method includes: Acquire multiple historical voice inputs from the interactive object to the digital human object; Based on a multi-dimensional vector extraction algorithm, determine the multi-dimensional vectors corresponding to all the historical input speech; Based on the multi-dimensional vector, predict the contextual parameters corresponding to the interactive object and the digital human object; Based on the prediction model corresponding to the context parameters and the historical input speech, the interaction rules of the digital human object relative to the interactive object are determined; the interaction rules are used to limit the data rules followed by the response speech and / or response actions of the digital human object.

2. The digital human interaction method based on speech recognition according to claim 1, characterized in that, The multi-dimensional vector extraction algorithm determines the multi-dimensional vectors corresponding to all the historical input speech, including: For each historical input speech, the historical input speech is input into a multi-dimensional parameter prediction model to obtain the parameters of multiple dimensions corresponding to the historical input speech; Based on the vectorization model corresponding to each parameter, the historical input speech is vectorized to obtain the speech parameter vector; The speech parameter vectors of all the historical input speech are assembled to obtain the corresponding multi-dimensional vector.

3. The digital human interaction method based on speech recognition according to claim 2, characterized in that, The dimension is either a spatial dimension or a temporal dimension; the spatial dimension includes at least one of an echo dimension, a distance dimension, and a location dimension; the temporal dimension includes at least one of a speech rate dimension, a volume dimension, and a duration dimension.

4. The digital human interaction method based on speech recognition according to claim 2, characterized in that, The step of vectorizing the historical input speech according to the vectorization model corresponding to each parameter to obtain a speech parameter vector includes: For each parameter, the corresponding vectorized model is matched among multiple candidate vectorized models; The historical input speech is input into the vectorized model to obtain the parameter-related vector corresponding to the parameter; Based on the preset parameter structure corresponding to all the parameters, the speech parameter vector corresponding to the historical input speech is obtained by assembling all the parameter-related vectors.

5. The digital human interaction method based on speech recognition according to claim 2, characterized in that, The step of assembling the speech parameter vectors of all the historical input speech to obtain the corresponding multi-dimensional vector includes: Calculate the vector similarity between the speech parameter vectors of any two historical input speech samples; Each historical input speech is defined as a graph node, and the corresponding speech parameter vector is defined as the node feature value of the graph node. The similarity of the vectors corresponding to any two graph nodes is determined as the node association feature between the two graph nodes; All the graph nodes and their corresponding node feature values ​​and node association features are determined as corresponding multi-dimensional vectors.

6. The digital human interaction method based on speech recognition according to claim 1, characterized in that, The step of predicting the contextual parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector includes: The multi-dimensional vectors are input into the trained context prediction model to obtain the context parameters corresponding to the interactive object and the digital human object; the context prediction model is trained using a training dataset that includes multiple training dialogue records and corresponding multi-dimensional vector annotations and context annotations.

7. The digital human interaction method based on speech recognition according to claim 1, characterized in that, The contextual parameters include at least one of the following: dialogue space type, dialogue space area, interaction purpose, interaction friendliness, and level of understanding.

8. The digital human interaction method based on speech recognition according to claim 1, characterized in that, The step of determining the interaction rules of the digital human object relative to the interactive object based on the prediction model corresponding to the context parameters and the historical input speech includes: Based on the correspondence between the scenario parameters and the preset first parameter, the motion restriction parameters of the digital human object are determined; the motion restriction parameters include at least one of amplitude restriction parameters, movement distance restriction parameters, swing angle restriction parameters, and movement part restriction parameters; Based on the correspondence between the context parameters and the preset second parameters, the voice restriction parameters of the digital human object are determined; the voice restriction parameters include at least one of the following: length restriction parameters, tone restriction parameters, voice-text restriction parameters, and voice volume restriction parameters. Generate a pre-cue word rule that includes the action restriction parameters and the speech restriction parameters; The pre-prompt word rule is connected to the data input interface of the multimodal model of the digital human object, so that each prediction data input is accompanied by a prompt word processed by the pre-prompt word rule to correct the output speech and / or output motion image of the multimodal model.

9. A digital human interaction system based on speech recognition, characterized in that, The system includes: The acquisition module is used to acquire multiple historical voice inputs from the interactive object to the digital human object; The extraction module is used to determine the multi-dimensional vectors corresponding to all the historical input speech based on a multi-dimensional vector extraction algorithm; The prediction module is used to predict the context parameters corresponding to the interactive object and the digital human object based on the multi-dimensional vector. The determination module is used to determine the interaction rules of the digital human object relative to the interactive object based on the prediction model corresponding to the context parameters and the historical input voice; the interaction rules are used to limit the data rules followed by the response voice and / or response actions of the digital human object.

10. A digital human interaction system based on speech recognition, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the digital human interaction method based on speech recognition as described in any one of claims 1-8.

Images